Bug: Deadlock in malloc on ARMv6 with uClibc-ng + Thumb1 + NPTL
Hello, We have noticed random deadlocks in malloc/free when using uClibc-ng + Thumb1 + NPTL compiled for ARMv6 in multi-threaded environment. Steps to reproduce 1. Build Buildroot 2019.02.9 SDK with the following configuration: BR2_arm=y BR2_arm1176jz_s=y BR2_ARM_INSTRUCTIONS_THUMB=y BR2_SHARED_STATIC_LIBS=y BR2_TOOLCHAIN_BUILDROOT_WCHAR=y BR2_TOOLCHAIN_BUILDROOT_USE_SSP=y BR2_TOOLCHAIN_BUILDROOT_CXX=y 2. Cross compile the sample (see below) application with the SDK using static linking: $ arm-buildroot-linux-uclibcgnueabi-gcc -static -g deadlock.c \ -lpthread -o deadlock 3. Run the application with at least two threads 3.1. Run the application using QEMU static emulation: $ qemu-arm-static ./deadlock 4 1000000 3.2. Run the application on a device with ARMv6 support (ex. any RPi with Raspbian image): $ ./deadlock 4 1000000 Expected results: The application should always exit cleanly. Actual results: The application often deadlocks in both cases. Notes: 1. The bug is also reproducible with latest uClibc-ng (v1.0.32) 2. According to our tests the endianness does not matter 3. The callstack when deadlock appears: (gdb) info threads Id Target Id Frame * 1 LWP 11653 "mutex" 0x000165b2 in pthread_join () 2 LWP 11656 "mutex" 0x00014cac in __lll_lock_wait () 3 LWP 11657 "mutex" 0x00014ce2 in __lll_lock_wait () 4 LWP 11658 "mutex" 0x00014cac in __lll_lock_wait () 5 LWP 11659 "mutex" 0x00014cac in __lll_lock_wait () (gdb) t 1 [Switching to thread 1 (LWP 11653)] #0 0x000165b2 in pthread_join () (gdb) bt #0 0x000165b2 in pthread_join () #1 0x00010396 in main (argc=3, argv=0x7efff784) at deadlock.c:42 (gdb) t 2 [Switching to thread 2 (LWP 11656)] #0 0x00014cac in __lll_lock_wait () (gdb) bt #0 0x00014cac in __lll_lock_wait () #1 0x00016b7e in pthread_mutex_lock () #2 0x0001199e in __libc_malloc () #3 0x00010278 in do_request (data=0x1) at deadlock.c:13 #4 0x00015b36 in start_thread () #5 0x000173f4 in clone () (gdb) t 3 [Switching to thread 3 (LWP 11657)] #0 0x00014ce2 in __lll_lock_wait () (gdb) bt #0 0x00014ce2 in __lll_lock_wait () #1 0x00016b7e in pthread_mutex_lock () #2 0x000117c2 in __libc_free () #3 0x00010284 in do_request (data=0x2) at deadlock.c:15 #4 0x00015b36 in start_thread () #5 0x000173f4 in clone () (gdb) t 4 [Switching to thread 4 (LWP 11658)] #0 0x00014cac in __lll_lock_wait () (gdb) bt #0 0x00014cac in __lll_lock_wait () #1 0x00016b7e in pthread_mutex_lock () #2 0x0001199e in __libc_malloc () #3 0x00010278 in do_request (data=0x3) at deadlock.c:13 #4 0x00015b36 in start_thread () #5 0x000173f4 in clone () (gdb) t 5 [Switching to thread 5 (LWP 11659)] #0 0x00014cac in __lll_lock_wait () (gdb) bt #0 0x00014cac in __lll_lock_wait () #1 0x00016b7e in pthread_mutex_lock () #2 0x0001199e in __libc_malloc () #3 0x00010278 in do_request (data=0x4) at deadlock.c:13 #4 0x00015b36 in start_thread () #5 0x000173f4 in clone () Sample application: #include <stdlib.h> #include <stdio.h> #include <pthread.h> int count; void *do_request(void *data) { for (int i = 0; i < count; i++) { if (i % 1000 == 0) printf("Alloc for %p count %d\n", data, i); char *buffer = malloc(32 * sizeof(char)); free(buffer); } return NULL; } int main(int argc, char *argv[]) { int i, rc; int threads_count = atoi(argv[1]); count = atoi(argv[2]); pthread_t threads[threads_count]; printf("Creating %d threads\n", threads_count); for (i = 0; i < threads_count; i++) { rc = pthread_create(&threads[i], NULL, do_request, (int*)(i+1)); if (rc) { printf("Error creating thread\n"); exit(1); } } for (i = 0; i < threads_count; i++) if (pthread_join(threads[i], NULL)) { printf("Error joining thread\n"); exit(1); } printf("Done. \n"); return 0; } Best regards, Gyula Farkas
Hi, Gyula FARKAS wrote,
Hello,
We have noticed random deadlocks in malloc/free when using uClibc-ng + Thumb1 + NPTL compiled for ARMv6 in multi-threaded environment.
Your example segfaults directly for me under qemu-system-arm or even on Debian/amd64 with gibc. Is it possible the code is broken? best regards Waldemar
Steps to reproduce
1. Build Buildroot 2019.02.9 SDK with the following configuration:
BR2_arm=y BR2_arm1176jz_s=y BR2_ARM_INSTRUCTIONS_THUMB=y BR2_SHARED_STATIC_LIBS=y BR2_TOOLCHAIN_BUILDROOT_WCHAR=y BR2_TOOLCHAIN_BUILDROOT_USE_SSP=y BR2_TOOLCHAIN_BUILDROOT_CXX=y
2. Cross compile the sample (see below) application with the SDK using static linking:
$ arm-buildroot-linux-uclibcgnueabi-gcc -static -g deadlock.c \ -lpthread -o deadlock
3. Run the application with at least two threads
3.1. Run the application using QEMU static emulation:
$ qemu-arm-static ./deadlock 4 1000000
3.2. Run the application on a device with ARMv6 support (ex. any RPi with Raspbian image):
$ ./deadlock 4 1000000
Expected results:
The application should always exit cleanly.
Actual results:
The application often deadlocks in both cases.
Notes:
1. The bug is also reproducible with latest uClibc-ng (v1.0.32)
2. According to our tests the endianness does not matter
3. The callstack when deadlock appears:
(gdb) info threads Id Target Id Frame * 1 LWP 11653 "mutex" 0x000165b2 in pthread_join () 2 LWP 11656 "mutex" 0x00014cac in __lll_lock_wait () 3 LWP 11657 "mutex" 0x00014ce2 in __lll_lock_wait () 4 LWP 11658 "mutex" 0x00014cac in __lll_lock_wait () 5 LWP 11659 "mutex" 0x00014cac in __lll_lock_wait ()
(gdb) t 1 [Switching to thread 1 (LWP 11653)] #0 0x000165b2 in pthread_join ()
(gdb) bt #0 0x000165b2 in pthread_join () #1 0x00010396 in main (argc=3, argv=0x7efff784) at deadlock.c:42
(gdb) t 2 [Switching to thread 2 (LWP 11656)] #0 0x00014cac in __lll_lock_wait ()
(gdb) bt #0 0x00014cac in __lll_lock_wait () #1 0x00016b7e in pthread_mutex_lock () #2 0x0001199e in __libc_malloc () #3 0x00010278 in do_request (data=0x1) at deadlock.c:13 #4 0x00015b36 in start_thread () #5 0x000173f4 in clone ()
(gdb) t 3 [Switching to thread 3 (LWP 11657)] #0 0x00014ce2 in __lll_lock_wait ()
(gdb) bt #0 0x00014ce2 in __lll_lock_wait () #1 0x00016b7e in pthread_mutex_lock () #2 0x000117c2 in __libc_free () #3 0x00010284 in do_request (data=0x2) at deadlock.c:15 #4 0x00015b36 in start_thread () #5 0x000173f4 in clone ()
(gdb) t 4 [Switching to thread 4 (LWP 11658)] #0 0x00014cac in __lll_lock_wait ()
(gdb) bt #0 0x00014cac in __lll_lock_wait () #1 0x00016b7e in pthread_mutex_lock () #2 0x0001199e in __libc_malloc () #3 0x00010278 in do_request (data=0x3) at deadlock.c:13 #4 0x00015b36 in start_thread () #5 0x000173f4 in clone ()
(gdb) t 5 [Switching to thread 5 (LWP 11659)] #0 0x00014cac in __lll_lock_wait ()
(gdb) bt #0 0x00014cac in __lll_lock_wait () #1 0x00016b7e in pthread_mutex_lock () #2 0x0001199e in __libc_malloc () #3 0x00010278 in do_request (data=0x4) at deadlock.c:13 #4 0x00015b36 in start_thread () #5 0x000173f4 in clone ()
Sample application:
#include <stdlib.h> #include <stdio.h> #include <pthread.h>
int count;
void *do_request(void *data) { for (int i = 0; i < count; i++) { if (i % 1000 == 0) printf("Alloc for %p count %d\n", data, i); char *buffer = malloc(32 * sizeof(char));
free(buffer); } return NULL; }
int main(int argc, char *argv[]) { int i, rc; int threads_count = atoi(argv[1]);
count = atoi(argv[2]);
pthread_t threads[threads_count]; printf("Creating %d threads\n", threads_count);
for (i = 0; i < threads_count; i++) { rc = pthread_create(&threads[i], NULL, do_request, (int*)(i+1));
if (rc) { printf("Error creating thread\n"); exit(1); } }
for (i = 0; i < threads_count; i++) if (pthread_join(threads[i], NULL)) { printf("Error joining thread\n"); exit(1); }
printf("Done. \n"); return 0; }
Best regards, Gyula Farkas _______________________________________________ devel mailing list devel@uclibc-ng.org https://mailman.uclibc-ng.org/cgi-bin/mailman/listinfo/devel
Hello, The segfault is from the missing command line arguments, which I did not check in sample app. Please run it as: $ ./deadlock 4 1000000 Best regards, Gyula Farkas ________________________________________ From: Waldemar Brodkorb <wbx@uclibc-ng.org> Sent: Thursday, February 20, 2020 3:44 PM To: Gyula FARKAS Cc: devel@uclibc-ng.org Subject: Re: [uclibc-ng-devel] Bug: Deadlock in malloc on ARMv6 with uClibc-ng + Thumb1 + NPTL Hi, Gyula FARKAS wrote,
Hello,
We have noticed random deadlocks in malloc/free when using uClibc-ng + Thumb1 + NPTL compiled for ARMv6 in multi-threaded environment.
Your example segfaults directly for me under qemu-system-arm or even on Debian/amd64 with gibc. Is it possible the code is broken? best regards Waldemar
Steps to reproduce
1. Build Buildroot 2019.02.9 SDK with the following configuration:
BR2_arm=y BR2_arm1176jz_s=y BR2_ARM_INSTRUCTIONS_THUMB=y BR2_SHARED_STATIC_LIBS=y BR2_TOOLCHAIN_BUILDROOT_WCHAR=y BR2_TOOLCHAIN_BUILDROOT_USE_SSP=y BR2_TOOLCHAIN_BUILDROOT_CXX=y
2. Cross compile the sample (see below) application with the SDK using static linking:
$ arm-buildroot-linux-uclibcgnueabi-gcc -static -g deadlock.c \ -lpthread -o deadlock
3. Run the application with at least two threads
3.1. Run the application using QEMU static emulation:
$ qemu-arm-static ./deadlock 4 1000000
3.2. Run the application on a device with ARMv6 support (ex. any RPi with Raspbian image):
$ ./deadlock 4 1000000
Expected results:
The application should always exit cleanly.
Actual results:
The application often deadlocks in both cases.
Notes:
1. The bug is also reproducible with latest uClibc-ng (v1.0.32)
2. According to our tests the endianness does not matter
3. The callstack when deadlock appears:
(gdb) info threads Id Target Id Frame * 1 LWP 11653 "mutex" 0x000165b2 in pthread_join () 2 LWP 11656 "mutex" 0x00014cac in __lll_lock_wait () 3 LWP 11657 "mutex" 0x00014ce2 in __lll_lock_wait () 4 LWP 11658 "mutex" 0x00014cac in __lll_lock_wait () 5 LWP 11659 "mutex" 0x00014cac in __lll_lock_wait ()
(gdb) t 1 [Switching to thread 1 (LWP 11653)] #0 0x000165b2 in pthread_join ()
(gdb) bt #0 0x000165b2 in pthread_join () #1 0x00010396 in main (argc=3, argv=0x7efff784) at deadlock.c:42
(gdb) t 2 [Switching to thread 2 (LWP 11656)] #0 0x00014cac in __lll_lock_wait ()
(gdb) bt #0 0x00014cac in __lll_lock_wait () #1 0x00016b7e in pthread_mutex_lock () #2 0x0001199e in __libc_malloc () #3 0x00010278 in do_request (data=0x1) at deadlock.c:13 #4 0x00015b36 in start_thread () #5 0x000173f4 in clone ()
(gdb) t 3 [Switching to thread 3 (LWP 11657)] #0 0x00014ce2 in __lll_lock_wait ()
(gdb) bt #0 0x00014ce2 in __lll_lock_wait () #1 0x00016b7e in pthread_mutex_lock () #2 0x000117c2 in __libc_free () #3 0x00010284 in do_request (data=0x2) at deadlock.c:15 #4 0x00015b36 in start_thread () #5 0x000173f4 in clone ()
(gdb) t 4 [Switching to thread 4 (LWP 11658)] #0 0x00014cac in __lll_lock_wait ()
(gdb) bt #0 0x00014cac in __lll_lock_wait () #1 0x00016b7e in pthread_mutex_lock () #2 0x0001199e in __libc_malloc () #3 0x00010278 in do_request (data=0x3) at deadlock.c:13 #4 0x00015b36 in start_thread () #5 0x000173f4 in clone ()
(gdb) t 5 [Switching to thread 5 (LWP 11659)] #0 0x00014cac in __lll_lock_wait ()
(gdb) bt #0 0x00014cac in __lll_lock_wait () #1 0x00016b7e in pthread_mutex_lock () #2 0x0001199e in __libc_malloc () #3 0x00010278 in do_request (data=0x4) at deadlock.c:13 #4 0x00015b36 in start_thread () #5 0x000173f4 in clone ()
Sample application:
#include <stdlib.h> #include <stdio.h> #include <pthread.h>
int count;
void *do_request(void *data) { for (int i = 0; i < count; i++) { if (i % 1000 == 0) printf("Alloc for %p count %d\n", data, i); char *buffer = malloc(32 * sizeof(char));
free(buffer); } return NULL; }
int main(int argc, char *argv[]) { int i, rc; int threads_count = atoi(argv[1]);
count = atoi(argv[2]);
pthread_t threads[threads_count]; printf("Creating %d threads\n", threads_count);
for (i = 0; i < threads_count; i++) { rc = pthread_create(&threads[i], NULL, do_request, (int*)(i+1));
if (rc) { printf("Error creating thread\n"); exit(1); } }
for (i = 0; i < threads_count; i++) if (pthread_join(threads[i], NULL)) { printf("Error joining thread\n"); exit(1); }
printf("Done. \n"); return 0; }
Best regards, Gyula Farkas _______________________________________________ devel mailing list devel@uclibc-ng.org https://mailman.uclibc-ng.org/cgi-bin/mailman/listinfo/devel
________________________ This email was scanned by Bitdefender
Hello, We have managed to reproduce the issue with your binary. Tested with (./deadlock 4 1000000): - Raspberry Pi 1 with Raspbian Buster - Raspberry Pi 3 with Raspbian Buster - QEMU 4.2.0 on a Core i7-8650U Archlinux - QEMU 4.2.0 on a Core i7-8700 Gentoo Maybe you are using an older version of QEMU which does not have support for Multi-Threaded TCG? One single core machines/emulation the mutex deadlocks much less often. The occurrence depends on CPU speed and thread numbers. You may need to increase the thread numbers (first argument) and iteration count (second argument) or even try to run several times. Best regards, Gyula FARKAS On 2/21/20 1:30 PM, Waldemar Brodkorb wrote:
Hi, Gyula FARKAS wrote,
Hello,
The segfault is from the missing command line arguments, which I did not check in sample app. Please run it as:
$ ./deadlock 4 1000000
Does it happen with the attached binary? I couldn't reproduce in Qemu.
best regards Waldemar
________________________ This email was scanned by Bitdefender
participants (2)
-
Gyula FARKAS -
Waldemar Brodkorb