Hi,
After I upgraded uClibc-ng from 1.0.9 to 1.0.10 for MIPSEL, I get some segfaults that didn't happen before the upgrade. And this happens with 1.0.11-1.0.12 too. I attached transmission 2.84 gdb crash log. The crash happens some while after a torrent is initiated. I tested this with 1.0.10-1.0.11 too (the log is from 1.0.12), and got the same issue. If I downgrade to 1.0.9, transmission works OK.
There's something wrong for other programs too, e.g., deluge keeps spawning 'sslv3 alert bad certificate' error during daemon-client handshake. pyload crashes as well (not tested by me, but I have user reports). ARMv7 uClibc-ng 1.0.12 platform doesn't have these issues.
Let me know if there's more info that you need.
Thanks, Alex
On Mon, Feb 15, 2016 at 5:28 PM, Alex Potapenko opotapenko@gmail.com wrote:
Hi,
After I upgraded uClibc-ng from 1.0.9 to 1.0.10 for MIPSEL, I get some segfaults that didn't happen before the upgrade.
Which MIPS version you are using, exactly? i.e. mips32r2? mips64r6? Could you provide your .config?
ARMv7 uClibc-ng 1.0.12 platform doesn't have these issues.
Are you able to bisect commits between 1.0.9 - 1.0.10 ? The single MIPS-specific changes, I can see, is - "Replace MIPS specific memcpy.S/memset.S with version from glibc/newlib." 2636b17616
Regards, Leonid
Hi Leonid!
2016-02-15 16:59 GMT+02:00 Leonid Lisovskiy lly.dev@gmail.com:
Which MIPS version you are using, exactly? i.e. mips32r2? mips64r6? Could you provide your .config?
mips32r2. The config is the default buildroot config (except for prefix and locales): see attached.
Are you able to bisect commits between 1.0.9 - 1.0.10 ? The single MIPS-specific changes, I can see, is - "Replace MIPS specific memcpy.S/memset.S with version from glibc/newlib." 2636b17616
Bingo! After I reverted memcpy.S/memset.S to pre-2636b17616 (e3c3bf2b580), the issue with transmission is gone. So does the deluge sslv3 handshake issue I described.
I'll add patch to Optware-ng to revert memcpy.S/memset.S for now.
Thanks, Alex
Hi Alex, Alex Potapenko wrote,
Hi Leonid!
2016-02-15 16:59 GMT+02:00 Leonid Lisovskiy lly.dev@gmail.com:
Which MIPS version you are using, exactly? i.e. mips32r2? mips64r6? Could you provide your .config?
mips32r2. The config is the default buildroot config (except for prefix and locales): see attached.
Are you able to bisect commits between 1.0.9 - 1.0.10 ? The single MIPS-specific changes, I can see, is - "Replace MIPS specific memcpy.S/memset.S with version from glibc/newlib." 2636b17616
Bingo! After I reverted memcpy.S/memset.S to pre-2636b17616 (e3c3bf2b580), the issue with transmission is gone. So does the deluge sslv3 handshake issue I described.
I'll add patch to Optware-ng to revert memcpy.S/memset.S for now.
I still trying to reproduce the bug here. It seems the Debian torrent is loading fine on my RB532 MIPS32: transmission-daemon -a 10.* -f [1970-01-01 01:13:02.504 CET] Transmission 2.84 (14307) started (session.c:736) [1970-01-01 01:13:02.520 CET] RPC Server Adding address to whitelist: 10.* (rpc-server.c:824) [1970-01-01 01:13:02.524 CET] RPC Server Serving RPC and Web requests on port 127.0.0.1:9091/transmission/ (rpc-server.c:1033) [1970-01-01 01:13:02.529 CET] RPC Server Whitelist enabled (rpc-server.c:1037) [1970-01-01 01:13:02.533 CET] UDP Failed to set receive buffer: requested 4194304, got 311296 (tr-udp.c:78) [1970-01-01 01:13:02.539 CET] UDP Please add the line "net.core.rmem_max = 4194304" to /etc/sysctl.conf (tr-udp.c:83) [1970-01-01 01:13:02.543 CET] UDP Failed to set send buffer: requested 1048576, got 311296 (tr-udp.c:89) [1970-01-01 01:13:02.548 CET] UDP Please add the line "net.core.wmem_max = 1048576" to /etc/sysctl.conf (tr-udp.c:94) [1970-01-01 01:13:02.552 CET] DHT Generating new id (tr-dht.c:310) [1970-01-01 01:13:02.556 CET] Using settings from "/root/.config/transmission-daemon" (daemon.c:557) [1970-01-01 01:13:02.560 CET] Port Forwarding (NAT-PMP) initnatpmp succeeded (0) (natpmp.c:70) [1970-01-01 01:13:02.564 CET] Port Forwarding (NAT-PMP) sendpublicaddressrequest succeeded (2) (natpmp.c:70) [1970-01-01 01:13:02.568 CET] Saved "/root/.config/transmission-daemon/settings.json" (variant.c:1214) [1970-01-01 01:13:02.573 CET] Port Forwarding State changed from "Not forwarded" to "Starting" (port-forwarding.c:92) [1970-01-01 01:13:02.580 CET] Port Forwarding State changed from "Starting" to "???" (port-forwarding.c:92) [1970-01-01 01:13:23.504 CET] DHT Attempting bootstrap from dht.transmissionbt.com (tr-dht.c:248) [1970-01-01 01:13:27.504 CET] Searching for web interface file "/root/.local/share/transmission/web/index.html" (platform.c:405) [1970-01-01 01:13:27.508 CET] Searching for web interface file "/usr/share/transmission/web/index.html" (platform.c:405) [1970-01-01 01:14:39.504 CET] Saved "/root/.config/transmission-daemon/torrents/debian-8.3.0-amd64-CD-1.iso.c388a78f357c7023.torrent" (variant.c:1214) [1970-01-01 01:14:39.508 CET] debian-8.3.0-amd64-CD-1.iso Queued for verification (verify.c:264) [1970-01-01 01:14:39.512 CET] debian-8.3.0-amd64-CD-1.iso Verifying torrent (verify.c:219) [1970-01-01 01:14:39.516 CET] debian-8.3.0-amd64-CD-1.iso Starting IPv4 DHT announce (poor, 30 nodes) (tr-dht.c:576)
[1970-01-01 01:18:57.508 CET] Saved "/root/.config/transmission-daemon/resume/debian-8.3.0-amd64-CD-1.iso.c388a78f357c7023.resume" (variant.c:1214) [1970-01-01 01:18:57.514 CET] Saved "/root/.config/transmission-daemon/stats.json" (variant.c:1214)
Is your toolchain soft-float or hard-float?
best regards Waldemar
Hi Waldemar!
2016-02-17 0:34 GMT+02:00 Waldemar Brodkorb wbx@uclibc-ng.org:
I still trying to reproduce the bug here. It seems the Debian torrent is loading fine on my RB532 MIPS32: From what I know, on Optware-ng, before I unrolled the commit, any
torrent crashed few minutes after the launch.
Is your toolchain soft-float or hard-float?
Soft-float mips32r2 gcc 5.2.0 binutils 2.25.1. And it's soft-float mips32r2 gcc 4.8.5 binutils 2.24 that Entware-ng project uses, and the issues they experienced after upgrade were also gone after unrolling. Is yours hard-float? Maybe, it affects soft-float only.
Thanks, Alex
Hi Alex, Alex Potapenko wrote,
Hi Waldemar!
2016-02-17 0:34 GMT+02:00 Waldemar Brodkorb wbx@uclibc-ng.org:
I still trying to reproduce the bug here. It seems the Debian torrent is loading fine on my RB532 MIPS32:
From what I know, on Optware-ng, before I unrolled the commit, any torrent crashed few minutes after the launch.
Is your toolchain soft-float or hard-float?
Soft-float mips32r2 gcc 5.2.0 binutils 2.25.1. And it's soft-float mips32r2 gcc 4.8.5 binutils 2.24 that Entware-ng project uses, and the issues they experienced after upgrade were also gone after unrolling. Is yours hard-float? Maybe, it affects soft-float only.
I am still unable to reproduce using gcc 5.3.0 and binutils 2.25.1 with a recent 4.1.x Linux kernel. I tried on Mikrotik RB532 with soft and hard float. I tried with Qemu MIPS32r2 with soft and hard float.
Can you reproduce the transmission failure in Qemu? On what hardware the application fails?
best regards Waldemar
Hi Alex, Alex Potapenko wrote,
Hi Waldemar!
2016-02-17 0:34 GMT+02:00 Waldemar Brodkorb wbx@uclibc-ng.org:
I still trying to reproduce the bug here. It seems the Debian torrent is loading fine on my RB532 MIPS32:
From what I know, on Optware-ng, before I unrolled the commit, any torrent crashed few minutes after the launch.
Is your toolchain soft-float or hard-float?
Soft-float mips32r2 gcc 5.2.0 binutils 2.25.1. And it's soft-float mips32r2 gcc 4.8.5 binutils 2.24 that Entware-ng project uses, and the issues they experienced after upgrade were also gone after unrolling. Is yours hard-float? Maybe, it affects soft-float only.
Could you try this static linked binary including debug symbols for mips32r2 soft-float: http://downloads.uclibc-ng.org/temp/
Does it fail to download: http://cdimage.debian.org/debian-cd/8.3.0/amd64/bt-cd/debian-8.3.0-amd64-CD-...
Thanks Waldemar
Hi Waldemar!
2016-02-21 21:23 GMT+02:00 Waldemar Brodkorb wbx@uclibc-ng.org:
Could you try this static linked binary including debug symbols for mips32r2 soft-float: http://downloads.uclibc-ng.org/temp/
Does it fail to download: http://cdimage.debian.org/debian-cd/8.3.0/amd64/bt-cd/debian-8.3.0-amd64-CD-...
Yes, it crashes with segfault few minutes after torrent launch. Sorry, I don't have the time right now to try to reproduce it using qemu, since I'm currently busy with Optware-ng moving to Nas-Admin.org project infrastructure, but I assume this problem can be kernel-specific. I know that recent glibc require newer 3.x kernel, maybe, this issue happens with older kernels, so it makes sense to try 2.6 kernel with qemu (maybe, using buildroot to generate rootfs). You could start with 2.6.32.70 (if I remember right, this is the oldest official kernel supported by uClibc-ng). My router (and many other mipsel routers) runs a 2.6.22.19 kernel, here's custom 2.6.22.19 that can be used to build uCllibc-ng: https://github.com/wl500g/wl500g/tree/master/linux/linux-2.6.22.19
Hope this helps, Alex
Hi Alex, Alex Potapenko wrote,
Hi Waldemar!
2016-02-21 21:23 GMT+02:00 Waldemar Brodkorb wbx@uclibc-ng.org:
Could you try this static linked binary including debug symbols for mips32r2 soft-float: http://downloads.uclibc-ng.org/temp/
Does it fail to download: http://cdimage.debian.org/debian-cd/8.3.0/amd64/bt-cd/debian-8.3.0-amd64-CD-...
Yes, it crashes with segfault few minutes after torrent launch. Sorry, I don't have the time right now to try to reproduce it using qemu, since I'm currently busy with Optware-ng moving to Nas-Admin.org project infrastructure, but I assume this problem can be kernel-specific. I know that recent glibc require newer 3.x kernel, maybe, this issue happens with older kernels, so it makes sense to try 2.6 kernel with qemu (maybe, using buildroot to generate rootfs). You could start with 2.6.32.70 (if I remember right, this is the oldest official kernel supported by uClibc-ng). My router (and many other mipsel routers) runs a 2.6.22.19 kernel, here's custom 2.6.22.19 that can be used to build uCllibc-ng: https://github.com/wl500g/wl500g/tree/master/linux/linux-2.6.22.19
Okay, will try. Can you generate a full backtrace, uClibc-ng symbols are included, which where missing in your last backtrace. What hardware it is crashing on?
best regards Waldemar
2016-02-21 21:53 GMT+02:00 Waldemar Brodkorb wbx@uclibc-ng.org:
Okay, will try. Can you generate a full backtrace, uClibc-ng symbols are included, which where missing in your last backtrace. What hardware it is crashing on?
See attached info.txt with `cat /proc/cpuinfo` and gdb log
Best regards, Alex
Hi Alex, Alex Potapenko wrote,
2016-02-21 21:53 GMT+02:00 Waldemar Brodkorb wbx@uclibc-ng.org:
Okay, will try. Can you generate a full backtrace, uClibc-ng symbols are included, which where missing in your last backtrace. What hardware it is crashing on?
See attached info.txt with `cat /proc/cpuinfo` and gdb log
Just for completeness, can you provide /opt/etc/transmission-daemon?
best regards Waldemar
2016-02-21 22:10 GMT+02:00 Waldemar Brodkorb wbx@uclibc-ng.org:
Just for completeness, can you provide /opt/etc/transmission-daemon?
Sure, see attached
Best regards, Alex
Hi Alex, Alex Potapenko wrote,
2016-02-21 22:10 GMT+02:00 Waldemar Brodkorb wbx@uclibc-ng.org:
Just for completeness, can you provide /opt/etc/transmission-daemon?
Sure, see attached
Sorry, one last thing please: (gdb) handle SIGPIPE nostop noprint pass (gdb) r ... crash ... (gdb) bt full
The SIGPIPE seems misleading. See https://trac.transmissionbt.com/ticket/4520
May be ignoring SIGPIPE will show the culprit of the crash.
best regards Waldemar
2016-02-21 22:27 GMT+02:00 Waldemar Brodkorb wbx@uclibc-ng.org:
Sorry, one last thing please: (gdb) handle SIGPIPE nostop noprint pass (gdb) r ... crash ... (gdb) bt full
The SIGPIPE seems misleading. See https://trac.transmissionbt.com/ticket/4520
May be ignoring SIGPIPE will show the culprit of the crash.
See info2.txt
Best regards, Alex
-
Alex Potapenko -
Leonid Lisovskiy -
Waldemar Brodkorb