Bug: Deadlock in malloc on ARMv6 with uClibc-ng + Thumb1 + NPTL - devel

18 Feb 2020


      Hello,
We have noticed random deadlocks in malloc/free when using uClibc-ng +
Thumb1 + NPTL compiled for ARMv6 in multi-threaded environment.
Steps to reproduce
1. Build Buildroot 2019.02.9 SDK with the following configuration:
BR2_arm=y
     BR2_arm1176jz_s=y
     BR2_ARM_INSTRUCTIONS_THUMB=y
     BR2_SHARED_STATIC_LIBS=y
     BR2_TOOLCHAIN_BUILDROOT_WCHAR=y
     BR2_TOOLCHAIN_BUILDROOT_USE_SSP=y
     BR2_TOOLCHAIN_BUILDROOT_CXX=y
2. Cross compile the sample (see below) application with the SDK using
    static linking:
$ arm-buildroot-linux-uclibcgnueabi-gcc -static -g deadlock.c \
     -lpthread -o deadlock
3. Run the application with at least two threads
3.1. Run the application using QEMU static emulation:
$ qemu-arm-static ./deadlock 4 1000000
3.2. Run the application on a device with ARMv6 support (ex. any RPi
      with Raspbian image):
$ ./deadlock 4 1000000
Expected results:
The application should always exit cleanly.
Actual results:
The application often deadlocks in both cases.
Notes:
1. The bug is also reproducible with latest uClibc-ng (v1.0.32)
2. According to our tests the endianness does not matter
3. The callstack when deadlock appears:
(gdb) info threads
   Id   Target Id         Frame
* 1    LWP 11653 "mutex" 0x000165b2 in pthread_join ()
   2    LWP 11656 "mutex" 0x00014cac in __lll_lock_wait ()
   3    LWP 11657 "mutex" 0x00014ce2 in __lll_lock_wait ()
   4    LWP 11658 "mutex" 0x00014cac in __lll_lock_wait ()
   5    LWP 11659 "mutex" 0x00014cac in __lll_lock_wait ()
(gdb) t 1
[Switching to thread 1 (LWP 11653)]
#0  0x000165b2 in pthread_join ()
(gdb) bt
#0  0x000165b2 in pthread_join ()
#1  0x00010396 in main (argc=3, argv=0x7efff784) at deadlock.c:42
(gdb) t 2
[Switching to thread 2 (LWP 11656)]
#0  0x00014cac in __lll_lock_wait ()
(gdb) bt
#0  0x00014cac in __lll_lock_wait ()
#1  0x00016b7e in pthread_mutex_lock ()
#2  0x0001199e in __libc_malloc ()
#3  0x00010278 in do_request (data=0x1) at deadlock.c:13
#4  0x00015b36 in start_thread ()
#5  0x000173f4 in clone ()
(gdb) t 3
[Switching to thread 3 (LWP 11657)]
#0  0x00014ce2 in __lll_lock_wait ()
(gdb) bt
#0  0x00014ce2 in __lll_lock_wait ()
#1  0x00016b7e in pthread_mutex_lock ()
#2  0x000117c2 in __libc_free ()
#3  0x00010284 in do_request (data=0x2) at deadlock.c:15
#4  0x00015b36 in start_thread ()
#5  0x000173f4 in clone ()
(gdb) t 4
[Switching to thread 4 (LWP 11658)]
#0  0x00014cac in __lll_lock_wait ()
(gdb) bt
#0  0x00014cac in __lll_lock_wait ()
#1  0x00016b7e in pthread_mutex_lock ()
#2  0x0001199e in __libc_malloc ()
#3  0x00010278 in do_request (data=0x3) at deadlock.c:13
#4  0x00015b36 in start_thread ()
#5  0x000173f4 in clone ()
(gdb) t 5
[Switching to thread 5 (LWP 11659)]
#0  0x00014cac in __lll_lock_wait ()
(gdb) bt
#0  0x00014cac in __lll_lock_wait ()
#1  0x00016b7e in pthread_mutex_lock ()
#2  0x0001199e in __libc_malloc ()
#3  0x00010278 in do_request (data=0x4) at deadlock.c:13
#4  0x00015b36 in start_thread ()
#5  0x000173f4 in clone ()
Sample application:
#include <stdlib.h>
#include <stdio.h>
#include <pthread.h>
int count;
void *do_request(void *data)
{
     for (int i = 0; i < count; i++) {
         if (i % 1000 == 0)
             printf("Alloc for %p count %d\n", data, i);
         char *buffer = malloc(32 * sizeof(char));
free(buffer);
     }
     return NULL;
}
int main(int argc, char *argv[])
{
     int i, rc;
     int threads_count = atoi(argv[1]);
count = atoi(argv[2]);
pthread_t threads[threads_count];
     printf("Creating %d threads\n", threads_count);
for (i = 0; i < threads_count; i++) {
         rc = pthread_create(&threads[i], NULL, do_request, (int*)(i+1));
if (rc) {
             printf("Error creating thread\n");
             exit(1);
         }
     }
for (i = 0; i < threads_count; i++)
         if (pthread_join(threads[i], NULL)) {
             printf("Error joining thread\n");
             exit(1);
         }
printf("Done. \n");
     return 0;
}
Best regards,
Gyula Farkas