I have found a bug in .../libc/string/generic/memmove.c, which is the one that MIPS uses, since there's no specialized, optimized version for MIPS.
We're currently using uClibc v. 1.0.12, but I suspect the bug to be present in earlier releases too.
Here's a snippet from memmove.c#memmove(): ---------------------oOo--------------------- /* This test makes the forward copying code be used whenever possible. Reduces the working set. */ if (dstp - srcp >= len) /* *Unsigned* compare! */ { #ifndef __ARCH_HAS_BWD_MEMCPY__ /* Backward memcpy implementation cannot be used */ memcpy(dest, src, len); #else /* Copy from the beginning to the end. */ ---------------------oOo---------------------
Given the name of the define (__ARCH_HAS_BWD_MEMCPY__) it sounds as when this is defined, the architecture indeed has backward memcpy() support. But how come the line is preceded by #ifndef and not #ifdef, when the code inside calls memcpy()?
Also, the first comment inside the #ifndef seems odd, since memcpy() indeed is called: /* Backward memcpy implementation cannot be used */
Our SDK does not define __ARCH_HAS_BWD_MEMCPY__, so when memmove() resorts to a simple memcpy() that does the wrong thing for overlapping regions, our application fails with disastrous side-effects.
I have attached a patch that fixes this.
Please CC me in case of any inquiries/replies: rene.nielsen (at) microsemi.com
Thanks & regards, Rene Schipp von Branitz Nielsen
Hi Rene, Rene Nielsen wrote,
I have found a bug in .../libc/string/generic/memmove.c, which is the one that MIPS uses, since there's no specialized, optimized version for MIPS.
We're currently using uClibc v. 1.0.12, but I suspect the bug to be present in earlier releases too.
Here's a snippet from memmove.c#memmove(): ---------------------oOo--------------------- /* This test makes the forward copying code be used whenever possible. Reduces the working set. */ if (dstp - srcp >= len) /* *Unsigned* compare! */ { #ifndef __ARCH_HAS_BWD_MEMCPY__ /* Backward memcpy implementation cannot be used */ memcpy(dest, src, len); #else /* Copy from the beginning to the end. */ ---------------------oOo---------------------
Given the name of the define (__ARCH_HAS_BWD_MEMCPY__) it sounds as when this is defined, the architecture indeed has backward memcpy() support. But how come the line is preceded by #ifndef and not #ifdef, when the code inside calls memcpy()?
Also, the first comment inside the #ifndef seems odd, since memcpy() indeed is called: /* Backward memcpy implementation cannot be used */
Our SDK does not define __ARCH_HAS_BWD_MEMCPY__, so when memmove() resorts to a simple memcpy() that does the wrong thing for overlapping regions, our application fails with disastrous side-effects.
I have attached a patch that fixes this.
Please CC me in case of any inquiries/replies: rene.nielsen (at) microsemi.com
You are right, there is no special MIPS version for memmove(). The __ARCH_HAS_BWD_MEMCPY__ symbols was added in 2008 by this commit e4f55f33f69fce85099dd5936cc74856aa1b453d. I would rather say, that a default memcpy is used and there was a change for MIPS recently. Can you either try 1.0.9 or revert 2636b17616a19d628c3dbc373ebae67ef6e2b1f6 from 1.0.14?
The change to add MIPSr6 support ported from glibc and submitted by Steve Ellcey sellcey@imgtec.com seems to trigger a problem with old kernels on MIPS32 systems. If you search the mailinglist we had some reports in the past, but I still couldn't reproduce the issue to report to Steve.
Can you create a small test program showing your problem? Something for the included test-suite would be really great.
What MIPS system do you use? MIPS32r2? little or big endian? hard or soft-float toolchain? Which kernel version?
thanks Waldemar
Thanks for your swift reply, Waldemar!
We're using a flavor of MIPS32r2 called MIPS 24KEc. It's running little-endian with soft-floats, kernel version 3.14.
I have attached a tiny program (test.c) that fails (see fail.txt) on uClibc v. 1.0.12 without the patch I attached to my previous post.
Reverting to v. 1.0.9 also solves the problem!
Thanks, René (rene.nielsen (at) microsemi.com)
-----Original Message----- From: Waldemar Brodkorb [mailto:wbx@uclibc-ng.org] Sent: 21. april 2016 20:34 To: Rene Nielsen rene.nielsen@microsemi.com Cc: devel@uclibc-ng.org Subject: Re: [uclibc-ng-devel] memmove() is failing on MIPS CPU
EXTERNAL EMAIL
Hi Rene, Rene Nielsen wrote,
I have found a bug in .../libc/string/generic/memmove.c, which is the one that MIPS uses, since there's no specialized, optimized version for MIPS.
We're currently using uClibc v. 1.0.12, but I suspect the bug to be present in earlier releases too.
Here's a snippet from memmove.c#memmove(): ---------------------oOo--------------------- /* This test makes the forward copying code be used whenever possible. Reduces the working set. */ if (dstp - srcp >= len) /* *Unsigned* compare! */ { #ifndef __ARCH_HAS_BWD_MEMCPY__ /* Backward memcpy implementation cannot be used */ memcpy(dest, src, len); #else /* Copy from the beginning to the end. */ ---------------------oOo---------------------
Given the name of the define (__ARCH_HAS_BWD_MEMCPY__) it sounds as when this is defined, the architecture indeed has backward memcpy() support. But how come the line is preceded by #ifndef and not #ifdef, when the code inside calls memcpy()?
Also, the first comment inside the #ifndef seems odd, since memcpy() indeed is called: /* Backward memcpy implementation cannot be used */
Our SDK does not define __ARCH_HAS_BWD_MEMCPY__, so when memmove() resorts to a simple memcpy() that does the wrong thing for overlapping regions, our application fails with disastrous side-effects.
I have attached a patch that fixes this.
Please CC me in case of any inquiries/replies: rene.nielsen (at) microsemi.com
You are right, there is no special MIPS version for memmove(). The __ARCH_HAS_BWD_MEMCPY__ symbols was added in 2008 by this commit e4f55f33f69fce85099dd5936cc74856aa1b453d. I would rather say, that a default memcpy is used and there was a change for MIPS recently. Can you either try 1.0.9 or revert from 1.0.14?
The change to add MIPSr6 support ported from glibc and submitted by Steve Ellcey sellcey@imgtec.com seems to trigger a problem with old kernels on MIPS32 systems. If you search the mailinglist we had some reports in the past, but I still couldn't reproduce the issue to report to Steve.
Can you create a small test program showing your problem? Something for the included test-suite would be really great.
What MIPS system do you use? MIPS32r2? little or big endian? hard or soft-float toolchain? Which kernel version?
thanks Waldemar
Hi Rene, Rene Nielsen wrote,
Thanks for your swift reply, Waldemar!
We're using a flavor of MIPS32r2 called MIPS 24KEc. It's running little-endian with soft-floats, kernel version 3.14.
I have attached a tiny program (test.c) that fails (see fail.txt) on uClibc v. 1.0.12 without the patch I attached to my previous post.
Reverting to v. 1.0.9 also solves the problem!
It is okay to apply your test with following header: # Copyright (C) 2016 Rene Nielsen rene.nielsen@microsemi.com # Licensed under the LGPL v2.1, see the file COPYING.LIB in this # tarball.
I couldn't reproduce the failure, yet. Tried mips32 little endian with soft-float and MIPS32r2 in Qemu. binutils 2.25.1/gcc 5.3.0 and Linux Kernel 4.4.6.
Which binutils/gcc version are you using? Do you remove the FPU emulation from the kernel?
I will try with older Kernels next as I am still unsure how to reproduce the failure.
best regards Waldemar
Thanks for taking this seriously! And yes, please apply the header, you mentioned, to the test.
Here's a list of the host and target packages we use for the rootfs and the kernel, respectively:
Host, rootfs (Linux XXX 3.13.0-78-generic #122-Ubuntu SMP Mon Feb 1 23:11:33 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux): autoconf-2.69 automake-1.15 binutils-2.24 buildroot-2016.05-git cmake-3.4.3 fakeroot_1.20.2 gcc-5.3.0 gmp-6.1.0 libtool-2.4.6 m4-1.4.17 mpc-1.0.3 mpfr-3.1.4 pkgconf-0.9.12 xz-5.2.2
Target, rootfs: busybox-1.24.1 dropbear-2015.71 hiawatha-10.0 iptables-1.6.0 libevent-2.0.22-stable libfcgi-2.4.0 libupnp-1.6.19 linux-headers-3.14.63 mbedtls-2.2.1 mtd-1.5.2 netsnmp-5.7.3 ntp-4.2.8p6 strace-4.11 uclibc-1.0.12 xz-embedded-20130513 zlib-1.2.8
Host, kernel: autoconf-2.69 automake-1.15 binutils-2.24 buildroot-2016.05-git cmake-3.4.3 fakeroot_1.20.2 gcc-5.3.0 gmp-6.1.0 kmod-22 libtool-2.4.6 lz4-r131 lzo-2.09 lzop-1.03 m4-1.4.17 mpc-1.0.3 mpfr-3.1.4 pkgconf-0.9.12 squashfs4.3 xz-5.2.2 zlib-1.2.8
Target, kernel: linux-headers-3.14.63 mbedtls-2.2.1 uclibc-1.0.12 zlib-1.2.8
And here's a list of selected settings from the .config for the kernel: CONFIG_MIPS=y CONFIG_RWSEM_GENERIC_SPINLOCK=y CONFIG_GENERIC_HWEIGHT=y CONFIG_GENERIC_CALIBRATE_DELAY=y CONFIG_SCHED_OMIT_FRAME_POINTER=y CONFIG_BOOT_RAW=y CONFIG_CEVT_R4K=y CONFIG_CSRC_R4K=y CONFIG_DMA_NONCOHERENT=y CONFIG_NEED_DMA_MAP_STATE=y CONFIG_CPU_LITTLE_ENDIAN=y CONFIG_SYS_SUPPORTS_BIG_ENDIAN=y CONFIG_SYS_SUPPORTS_LITTLE_ENDIAN=y CONFIG_IRQ_CPU=y CONFIG_MIPS_L1_CACHE_SHIFT=5 CONFIG_CPU_MIPS32_R2=y CONFIG_SYS_HAS_CPU_MIPS32_R2=y CONFIG_CPU_MIPS32=y CONFIG_CPU_MIPSR2=y CONFIG_SYS_SUPPORTS_32BIT_KERNEL=y CONFIG_CPU_SUPPORTS_32BIT_KERNEL=y CONFIG_HARDWARE_WATCHPOINTS=y CONFIG_32BIT=y CONFIG_PAGE_SIZE_4KB=y CONFIG_FORCE_MAX_ZONEORDER=11 CONFIG_CPU_HAS_PREFETCH=y CONFIG_CPU_GENERIC_DUMP_TLB=y CONFIG_CPU_R4K_FPU=y CONFIG_CPU_R4K_CACHE_TLB=y CONFIG_MIPS_MT_DISABLED=y CONFIG_CPU_HAS_SYNC=y CONFIG_CPU_SUPPORTS_HIGHMEM=y CONFIG_ARCH_FLATMEM_ENABLE=y CONFIG_FLATMEM=y CONFIG_FLAT_NODE_MEM_MAP=y CONFIG_HAVE_MEMBLOCK=y CONFIG_HAVE_MEMBLOCK_NODE_MAP=y CONFIG_ARCH_DISCARD_MEMBLOCK=y CONFIG_PAGEFLAGS_EXTENDED=y CONFIG_SPLIT_PTLOCK_CPUS=4 CONFIG_COMPACTION=y CONFIG_MIGRATION=y CONFIG_ZONE_DMA_FLAG=0 CONFIG_VIRT_TO_BUS=y CONFIG_DEFAULT_MMAP_MIN_ADDR=4096 CONFIG_CROSS_MEMORY_ATTACH=y CONFIG_NEED_PER_CPU_KM=y CONFIG_HZ_250=y CONFIG_SYS_SUPPORTS_ARBIT_HZ=y CONFIG_HZ=250 CONFIG_PREEMPT_NONE=y CONFIG_SECCOMP=y CONFIG_LOCKDEP_SUPPORT=y CONFIG_STACKTRACE_SUPPORT=y CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config" CONFIG_IRQ_WORK=y CONFIG_BUILDTIME_EXTABLE_SORT=y
Please don't hesitate to write me, if you need additional info.
Thanks, René
-----Original Message----- From: Waldemar Brodkorb [mailto:wbx@uclibc-ng.org] Sent: 26. april 2016 08:48 To: Rene Nielsen rene.nielsen@microsemi.com Cc: Waldemar Brodkorb wbx@uclibc-ng.org; devel@uclibc-ng.org Subject: Re: [uclibc-ng-devel] memmove() is failing on MIPS CPU
EXTERNAL EMAIL
Hi Rene, Rene Nielsen wrote,
Thanks for your swift reply, Waldemar!
We're using a flavor of MIPS32r2 called MIPS 24KEc. It's running little-endian with soft-floats, kernel version 3.14.
I have attached a tiny program (test.c) that fails (see fail.txt) on uClibc v. 1.0.12 without the patch I attached to my previous post.
Reverting to v. 1.0.9 also solves the problem!
It is okay to apply your test with following header: # Copyright (C) 2016 Rene Nielsen rene.nielsen@microsemi.com # Licensed under the LGPL v2.1, see the file COPYING.LIB in this # tarball.
I couldn't reproduce the failure, yet. Tried mips32 little endian with soft-float and MIPS32r2 in Qemu. binutils 2.25.1/gcc 5.3.0 and Linux Kernel 4.4.6.
Which binutils/gcc version are you using? Do you remove the FPU emulation from the kernel?
I will try with older Kernels next as I am still unsure how to reproduce the failure.
best regards Waldemar
I've also experienced the same issues described by Rene. Up until now I've been applying the patches that entware-ng/optware-ng use which revert the mipsel changes between 1.0.9 and 1.0.10.
Without Rene's patch, or reverting those mipsel commits, Rene's test.c fails to run, and experiences same error's he experiences. With his patch it is working as expected.
Note: with or without patching, the static "test" binary I make works with qemu on x64_86 machine. It's only when I run it on the actual mipsel hardware that it fails. So testing with qemu for whatever reason is not an indication in this case of how it will work on the actual hardware.
kernel 2.6.22.19 binutils 2.25.1 gcc 5.3.0 mips32 soft-float
I'm working with routers like entware-ng/optware-ng, so am also stuck on the same kernel they are. I know entware-ng is using cross-tools-ng for their toolchain. I am using buildroot for mine.
thanks, Lance
On Thu, Apr 21, 2016 at 4:30 PM, Rene Nielsen rene.nielsen@microsemi.com wrote:
I have found a bug in .../libc/string/generic/memmove.c, which is the one that MIPS uses, since there's no specialized, optimized version for MIPS.
We're currently using uClibc v. 1.0.12, but I suspect the bug to be present in earlier releases too.
Here's a snippet from memmove.c#memmove(): ---------------------oOo--------------------- /* This test makes the forward copying code be used whenever possible. Reduces the working set. */ if (dstp - srcp >= len) /* *Unsigned* compare! */ { #ifndef __ARCH_HAS_BWD_MEMCPY__ /* Backward memcpy implementation cannot be used */ memcpy(dest, src, len); #else /* Copy from the beginning to the end. */ ---------------------oOo---------------------
Given the name of the define (__ARCH_HAS_BWD_MEMCPY__) it sounds as when this is defined, the architecture indeed has backward memcpy() support. But how come the line is preceded by #ifndef and not #ifdef, when the code inside calls memcpy()?
Also, the first comment inside the #ifndef seems odd, since memcpy() indeed is called: /* Backward memcpy implementation cannot be used */
Our SDK does not define __ARCH_HAS_BWD_MEMCPY__, so when memmove() resorts to a simple memcpy() that does the wrong thing for overlapping regions, our application fails with disastrous side-effects.
I have attached a patch that fixes this.
Hi Rene!
I am Optware-ng project maintainer. I have tried your patch to see if it fixes memset/memcpy related issue we were having on mipsel (e.g., see transmission issue https://github.com/Optware/Optware-ng/issues/53). Unfortunately, even though your test passes fine with 'memmove() succeeded', transmission still crashes after a while when a torrent is added, so I had to restore the old patch that reverted memcpy.S/memset.S to pre-2636b17616 (e3c3bf2b580), which fixed the issue. Probably, even with your patch, there's still something wrong with memcpy.S/memset.S.
Regards, Alex
Hi Alex, Alex Potapenko wrote,
On Thu, Apr 21, 2016 at 4:30 PM, Rene Nielsen rene.nielsen@microsemi.com wrote:
I have found a bug in .../libc/string/generic/memmove.c, which is the one that MIPS uses, since there's no specialized, optimized version for MIPS.
We're currently using uClibc v. 1.0.12, but I suspect the bug to be present in earlier releases too.
Here's a snippet from memmove.c#memmove(): ---------------------oOo--------------------- /* This test makes the forward copying code be used whenever possible. Reduces the working set. */ if (dstp - srcp >= len) /* *Unsigned* compare! */ { #ifndef __ARCH_HAS_BWD_MEMCPY__ /* Backward memcpy implementation cannot be used */ memcpy(dest, src, len); #else /* Copy from the beginning to the end. */ ---------------------oOo---------------------
Given the name of the define (__ARCH_HAS_BWD_MEMCPY__) it sounds as when this is defined, the architecture indeed has backward memcpy() support. But how come the line is preceded by #ifndef and not #ifdef, when the code inside calls memcpy()?
Also, the first comment inside the #ifndef seems odd, since memcpy() indeed is called: /* Backward memcpy implementation cannot be used */
Our SDK does not define __ARCH_HAS_BWD_MEMCPY__, so when memmove() resorts to a simple memcpy() that does the wrong thing for overlapping regions, our application fails with disastrous side-effects.
I have attached a patch that fixes this.
Hi Rene!
I am Optware-ng project maintainer. I have tried your patch to see if it fixes memset/memcpy related issue we were having on mipsel (e.g., see transmission issue https://github.com/Optware/Optware-ng/issues/53). Unfortunately, even though your test passes fine with 'memmove() succeeded', transmission still crashes after a while when a torrent is added, so I had to restore the old patch that reverted memcpy.S/memset.S to pre-2636b17616 (e3c3bf2b580), which fixed the issue. Probably, even with your patch, there's still something wrong with memcpy.S/memset.S.
Can you please try uClibc-ng 1.0.15 and disable UCLIBC_USE_MIPS_PREFETCH in your build?
We identified prefetch as the problematic instructions on the combination of MIPS CPU, kernel and uClibc-ng.
It is not perfect, but a valid workaround, until we find the reason for the problems.
Thanks in advance, Waldemar
Hi Waldemar!
On Sun, May 29, 2016 at 10:46 AM, Waldemar Brodkorb wbx@uclibc-ng.org wrote:
Can you please try uClibc-ng 1.0.15 and disable UCLIBC_USE_MIPS_PREFETCH in your build?
We identified prefetch as the problematic instructions on the combination of MIPS CPU, kernel and uClibc-ng.
It is not perfect, but a valid workaround, until we find the reason for the problems.
I confirm that disabling UCLIBC_USE_MIPS_PREFETCH in uClibc-ng 1.0.15 does the trick, so you're on the right track!
Thanks, Alex
-
Alex Potapenko -
Lance Fredrickson -
Rene Nielsen -
Waldemar Brodkorb