Hi Thomas, Thomas Petazzoni wrote,
Hello Waldemar,
We discussed a while ago the problem of supporting the static + PIE scenario, which currently causes some build failures of the Go-based flannel package, due to Scrt1.o being missing.
On IRC, you told me that Scrt1.o is properly produced by uClibc, and it looks like you suspected there was some issue in how Buildroot then packages the toolchain. So I looked at the build log of my toolchains, and here is what I found.
If I looked at the build log of the br-arm-full toolchain (an uClibc-ng based toolchain for ARM, with all features enabled: C++, locales, threads, wchar, etc.), I do see Scrt1.o being built:
AS lib/crt1.o AS lib/Scrt1.o AS lib/crti.o AS lib/crtn.o
Then installed:
install -m 644 ./lib/crt1.o ./lib/Scrt1.o ./lib/crti.o ./lib/crtn.o /opt/br-arm-full-2016.05-2-g5dabb45/usr/arm-buildroot-linux-uclibcgnueabi/sysroot/usr/lib//
And it is properly present in the toolchain tarball:
test@build:/opt$ tar tvf br-arm-full-2016.05-2-g5dabb45.tar.bz2 | grep crt -rw-r--r-- test/test 972 2016-06-01 14:27 br-arm-full-2016.05-2-g5dabb45/arm-buildroot-linux-uclibcgnueabi/sysroot/usr/lib/crtn.o -rw-r--r-- test/test 988 2016-06-01 14:27 br-arm-full-2016.05-2-g5dabb45/arm-buildroot-linux-uclibcgnueabi/sysroot/usr/lib/crti.o -rw-r--r-- test/test 1004 2016-06-01 14:27 br-arm-full-2016.05-2-g5dabb45/arm-buildroot-linux-uclibcgnueabi/sysroot/usr/lib/crt1.o -rw-r--r-- test/test 1080 2016-06-01 14:27 br-arm-full-2016.05-2-g5dabb45/arm-buildroot-linux-uclibcgnueabi/sysroot/usr/lib/Scrt1.o -rw-r--r-- test/test 1108 2016-06-01 14:31 br-arm-full-2016.05-2-g5dabb45/lib/gcc/arm-buildroot-linux-uclibcgnueabi/4.8.5/crtendS.o -rw-r--r-- test/test 1108 2016-06-01 14:31 br-arm-full-2016.05-2-g5dabb45/lib/gcc/arm-buildroot-linux-uclibcgnueabi/4.8.5/crtend.o -rw-r--r-- test/test 2528 2016-06-01 14:31 br-arm-full-2016.05-2-g5dabb45/lib/gcc/arm-buildroot-linux-uclibcgnueabi/4.8.5/crtbegin.o -rw-r--r-- test/test 2528 2016-06-01 14:31 br-arm-full-2016.05-2-g5dabb45/lib/gcc/arm-buildroot-linux-uclibcgnueabi/4.8.5/crtbeginT.o -rw-r--r-- test/test 2928 2016-06-01 14:31 br-arm-full-2016.05-2-g5dabb45/lib/gcc/arm-buildroot-linux-uclibcgnueabi/4.8.5/crtbeginS.o
However, the flannel failures don't happen with br-arm-full, but only with br-arm-full-static, which is the same toolchain, but built with BR2_STATIC_LIBS=y. Due to this, uClibc is built with HAVE_SHARED disabled.
And in this case, Scrt1.o is not built:
AS lib/crt1.o AS lib/crti.o AS lib/crtn.o
And it is not installed:
install -m 644 ./lib/crt1.o ./lib/crti.o ./lib/crtn.o /opt/br-arm-full-static-2016.05-2-g5dabb45/usr/arm-buildroot-linux-uclibcgnueabi/sysroot/usr/lib//
This explains why there is no Scrt1.o in the generated Buildroot toolchain.
So I continue to believe that the problem is on uClibc's side, and not on Buildroot's side.
It would be good if you could investigate why uClibc doesn't produce Scrt1.o when HAVE_SHARED is disabled.
At the moment Scrt1.o is only build under following condition: ifeq ($(HAVE_SHARED)$(UCLIBC_FORMAT_SHARED_FLAT),y)
I tried to compile a simple hello world as static PIE for ARM and this is not really straight forward. You need at least binutils 2.26 to use -W,--no-dynamic-linker otherwise PT_INTERP is added to the resulting executable. After that I tried to run it in Qemu (system and user level tried) and the binary fails to execute with a segmentation fault. So even if we add Scrt1.o somehow to the toolchains, the binaries will not work on the target. Scrt1.o is just PIC version of crt1.c. So the PIC assembly in uClibc-ng might be just broken.
Is it really required for flannel to work to use static PIE?
In my point of view it is some kind of security feature used on Linux together with address space layout randomization. I don't know flannel, but security related software as OpenSSH does allow to build with and without PIE.
I am not if and when I can take a look at the startup code. A short test for MIPS showed other problems and didn't compile my hello world as static PIE.
best regards Waldemar
Hello,
On Thu, 9 Jun 2016 23:01:30 +0200, Waldemar Brodkorb wrote:
It would be good if you could investigate why uClibc doesn't produce Scrt1.o when HAVE_SHARED is disabled.
At the moment Scrt1.o is only build under following condition: ifeq ($(HAVE_SHARED)$(UCLIBC_FORMAT_SHARED_FLAT),y)
I tried to compile a simple hello world as static PIE for ARM and this is not really straight forward. You need at least binutils 2.26 to use -W,--no-dynamic-linker otherwise PT_INTERP is added to the resulting executable. After that I tried to run it in Qemu (system and user level tried) and the binary fails to execute with a segmentation fault. So even if we add Scrt1.o somehow to the toolchains, the binaries will not work on the target. Scrt1.o is just PIC version of crt1.c. So the PIC assembly in uClibc-ng might be just broken.
Is it really required for flannel to work to use static PIE?
flannel is written in Go, so it's always built statically. So even when a dynamically-linked capable toolchain is used (which provides Scrt1.o), flannel is built as a static binary. Why would it work in this case, and not a fully static toolchain ?
In my point of view it is some kind of security feature used on Linux together with address space layout randomization. I don't know flannel, but security related software as OpenSSH does allow to build with and without PIE.
I agree that PIE should be optional, but I'm not sure if it's flannel itself that decides to be built with PIE, or just the whole Go stuff.
Thomas
Hello,
On Thu, 9 Jun 2016 23:10:05 +0200, Thomas Petazzoni wrote:
I agree that PIE should be optional, but I'm not sure if it's flannel itself that decides to be built with PIE, or just the whole Go stuff.
It's Go. I found again the piece of code that was pointed to me by Geoff. See https://golang.org/src/cmd/go/build.go, line 3188:
pie := (goarch == "arm" && goos == "linux") || goos == "android"
So for some reason, on ARM, PIE is forced enabled.
Thomas
Hi Thomas, Thomas Petazzoni wrote,
Hello,
On Thu, 9 Jun 2016 23:01:30 +0200, Waldemar Brodkorb wrote:
It would be good if you could investigate why uClibc doesn't produce Scrt1.o when HAVE_SHARED is disabled.
At the moment Scrt1.o is only build under following condition: ifeq ($(HAVE_SHARED)$(UCLIBC_FORMAT_SHARED_FLAT),y)
I tried to compile a simple hello world as static PIE for ARM and this is not really straight forward. You need at least binutils 2.26 to use -W,--no-dynamic-linker otherwise PT_INTERP is added to the resulting executable. After that I tried to run it in Qemu (system and user level tried) and the binary fails to execute with a segmentation fault. So even if we add Scrt1.o somehow to the toolchains, the binaries will not work on the target. Scrt1.o is just PIC version of crt1.c. So the PIC assembly in uClibc-ng might be just broken.
Is it really required for flannel to work to use static PIE?
flannel is written in Go, so it's always built statically. So even when a dynamically-linked capable toolchain is used (which provides Scrt1.o), flannel is built as a static binary. Why would it work in this case, and not a fully static toolchain ?
I mean "Why flannel requires static PIE?" In any case, if a fully static toolchain or a dynamic toolchain is used, the static PIE ARM binary will not be working until a ARM assembly expert make a fix.
In my point of view it is some kind of security feature used on Linux together with address space layout randomization. I don't know flannel, but security related software as OpenSSH does allow to build with and without PIE.
I agree that PIE should be optional, but I'm not sure if it's flannel itself that decides to be built with PIE, or just the whole Go stuff.
Okay. The short-term solution is to ask the go people to make pie optional.
best regards Waldemar
-
Thomas Petazzoni -
Waldemar Brodkorb