android NDK aarch64 cross-compilation fails while linking

[David Disseldorp]

I'm pretty new to Android development, here is the procedure I'm using to reproduce this:

1. Download NDK (https://dl.google.com/android/repository/android-ndk-r20-linux-x86_64.zip)
2. Generate standalone toolchain
   • ./make-standalone-toolchain.sh --arch=arm64 --verbose --platform=android-28 --install-dir=<dir>
   • add <dir>/bin/ to PATH
3. git clone https://github.com/lkl/linux (HEAD = 0a25540)
4. build with make CROSS_COMPILE=aarch64-linux-android- -C tools/lkl/ V=1

Compilation runs fine and successfully produces tools/lkl/liblkl.a, etc. however, the test binaries fail to link:

> make CROSS_COMPILE=aarch64-linux-android- -C tools/lkl/ V=1
make: Entering directory '/mnt/resource/linux/tools/lkl'
make -f /mnt/resource/linux/tools/build/Makefile.build dir=tests obj=boot
make -f /mnt/resource/linux/tools/build/Makefile.build dir=lib obj=liblkl
aarch64-linux-android-gcc -pie  -o /mnt/resource/linux/tools/lkl/tests/boot /mnt/resource/linux/tools/lkl/tests/boot-in.o /mnt/resource/linux/tools/lkl/liblkl.a  
/mnt/resource/linux/tools/lkl/liblkl.a(lkl.o): In function `printk_late_init':
/mnt/resource/linux/kernel/printk/printk.c:2899: undefined reference to `no symbol'
/mnt/resource/linux/kernel/printk/printk.c:2899: undefined reference to `no symbol'
/mnt/resource/linux/tools/lkl/liblkl.a(lkl.o): In function `mem_init_print_info':
/mnt/resource/linux/mm/page_alloc.c:7519: undefined reference to `no symbol'
/mnt/resource/linux/mm/page_alloc.c:7519: undefined reference to `no symbol'
/mnt/resource/linux/tools/lkl/liblkl.a(lkl.o): In function `free_initmem_default':
/mnt/resource/linux/./include/linux/mm.h:2147: undefined reference to `no symbol'
/mnt/resource/linux/tools/lkl/liblkl.a(lkl.o):/mnt/resource/linux/./include/linux/mm.h:2147: more undefined references to `no symbol' follow
clang80: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [Makefile:82: /mnt/resource/linux/tools/lkl/tests/boot] Error 1
make: Leaving directory '/mnt/resource/linux/tools/lkl'

Full invocation debug information:

> aarch64-linux-android-gcc -v -pie  -o /mnt/resource/linux/tools/lkl/tests/boot /mnt/resource/linux/tools/lkl/tests/boot-in.o /mnt/resource/linux/tools/lkl/liblkl.a
Android (5220042 based on r346389c) clang version 8.0.7 (https://android.googlesource.com/toolchain/clang b55f2d4ebfd35bf643d27dbca1bb228957008617) (https://android.googlesource.com/toolchain/llvm 3c393fe7a7e13b0fba4ac75a01aa683d7a5b11cd) (based on LLVM 8.0.7svn)
Target: aarch64-unknown-linux-android28
Thread model: posix
InstalledDir: /mnt/resource/aarch64-linux-android-api-28/bin
Found candidate GCC installation: /mnt/resource/aarch64-linux-android-api-28/bin/../lib/gcc/aarch64-linux-android/4.9.x
Selected GCC installation: /mnt/resource/aarch64-linux-android-api-28/bin/../lib/gcc/aarch64-linux-android/4.9.x
Candidate multilib: .;@m64
Selected multilib: .;@m64
 "/mnt/resource/aarch64-linux-android-api-28/bin/../lib/gcc/aarch64-linux-android/4.9.x/../../../../aarch64-linux-android/bin/ld" -pie -EL --fix-cortex-a53-843419 -z now -z relro --hash-style=gnu --enable-new-dtags --eh-frame-hdr -m aarch64linux -dynamic-linker /system/bin/linker64 -o /mnt/resource/linux/tools/lkl/tests/boot /mnt/resource/aarch64-linux-android-api-28/bin/../sysroot/usr/lib/aarch64-linux-android/28/crtbegin_dynamic.o -L/mnt/resource/aarch64-linux-android-api-28/lib64/clang/8.0.7/lib/linux/aarch64 -L/mnt/resource/aarch64-linux-android-api-28/bin/../lib/gcc/aarch64-linux-android/4.9.x -L/mnt/resource/aarch64-linux-android-api-28/bin/../lib/gcc/aarch64-linux-android/4.9.x/../../../../aarch64-linux-android/lib/../lib64 -L/mnt/resource/aarch64-linux-android-api-28/bin/../sysroot/usr/lib/aarch64-linux-android/28 -L/mnt/resource/aarch64-linux-android-api-28/bin/../sysroot/usr/lib/aarch64-linux-android -L/mnt/resource/aarch64-linux-android-api-28/bin/../lib/gcc/aarch64-linux-android/4.9.x/../../../../aarch64-linux-android/lib -L/mnt/resource/aarch64-linux-android-api-28/bin/../sysroot/usr/lib /mnt/resource/linux/tools/lkl/tests/boot-in.o /mnt/resource/linux/tools/lkl/liblkl.a -lgcc -ldl -lc -lgcc -ldl /mnt/resource/aarch64-linux-android-api-28/bin/../sysroot/usr/lib/aarch64-linux-android/28/crtend_android.o
/mnt/resource/linux/tools/lkl/liblkl.a(lkl.o): In function `printk_late_init':
/mnt/resource/linux/kernel/printk/printk.c:2899: undefined reference to `no symbol'
/mnt/resource/linux/kernel/printk/printk.c:2899: undefined reference to `no symbol'
/mnt/resource/linux/tools/lkl/liblkl.a(lkl.o): In function `mem_init_print_info':
/mnt/resource/linux/mm/page_alloc.c:7519: undefined reference to `no symbol'
/mnt/resource/linux/mm/page_alloc.c:7519: undefined reference to `no symbol'
/mnt/resource/linux/tools/lkl/liblkl.a(lkl.o): In function `free_initmem_default':
/mnt/resource/linux/./include/linux/mm.h:2147: undefined reference to `no symbol'
/mnt/resource/linux/tools/lkl/liblkl.a(lkl.o):/mnt/resource/linux/./include/linux/mm.h:2147: more undefined references to `no symbol' follow
clang80: error: linker command failed with exit code 1 (use -v to see invocation)

[Hajime Tazaki]

hmm, I have not tested newer android ndk recently so probably LKL might need something to follow up or work around those issues.

our test is based on relatively older one (ndk r15)

https://github.com/lkl/lkl-docker/blob/master/circleci/android/arm64/Dockerfile

[David Disseldorp]

hmm, I have not tested newer android ndk recently so probably LKL might need something to follow up or work around those issues.

our test is based on relatively older one (ndk r15)

https://github.com/lkl/lkl-docker/blob/master/circleci/android/arm64/Dockerfile

Thanks for the pointer. As an alternative to using the Linaro toolchain, I was able to get things working with NDK r21 (should also work with <= r20) by swapping out only aarch64-linux-android-ld for the stock openSUSE binary:

> /usr/bin/aarch64-suse-linux-ld -v
GNU ld (GNU Binutils; openSUSE Leap 15.1) 2.31.1.20180828-lp151.2
> home/ddiss/android-ndk-r21/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android-ld -v
GNU ld (GNU Binutils) 2.27.0.20170315 
> make AR=$TOOLCHAIN/bin/aarch64-linux-android-ar CC=$TOOLCHAIN/bin/aarch64-linux-android21-clang HOSTCC=gcc HOSTLD=ld KCFLAGS=-Wno-implicit-fallthrough LD=/usr/bin/aarch64-suse-linux-ld NM=$TOOLCHAIN/bin/aarch64-linux-android-nm OBJDUMP=$TOOLCHAIN/bin/aarch64-linux-android-objdump OBJSIZE=$TOOLCHAIN/bin/aarch64-linux-android-size OBJCOPY=$TOOLCHAIN/bin/aarch64-linux-android-objcopy -C tools/lkl/

I've no idea why openSUSE ld 2.31.1.20180828-lp151.2 works while NDK ld 2.27.0.20170315 fails, but I think this ticket can probably be closed as an NDK toolchain issue.