How to compile C++ program on Windows for ARM using LLVM?

Question

Aim Compile a C++ program on Windows for ARM using only LLVM.

Why LLVM because of permissive licensing.

I'm starting to wonder if my understanding of LLVM is correct.

On the host machine do

1. Use clang (front end) to generate intermediate representation. This representation is target independent.
2. Use llc (back end) to generate target assembly code.
3. Use lld-link.exe to produce executable.

Then execute on the target machine.

Host machine Windows 10, 64bit

Target machine Drive PX with a arm cortex-a57

The program

int main(int argc, char* argv[]) 
{
    int x=41;
    x++;
    return x;
}

I've checked out and compiled LLVM (using Visual Studio 2015, Release build, CPU= x64)

My attempts

clang.exe -target arm -march=armv8-a -mcpu=cortex-a57 -mfloat-abi=hard  -emit-llvm -c -o main.bc  main.cpp
llc.exe -march=arm -mcpu=cortex-a57 -mattr=a57,armv8-a,v8 -meabi=gnu -o main.s main.bc
lld-link.exe /entry:main /machine:arm main.s

Error

lld-link.exe: error: main.s: unknown file type

Then I tried doing the front-end steps on Windows and the back-end on the arm machine.

clang.exe -target arm -march=armv8-a -mcpu=cortex-a57 -mfloat-abi=hard  -emit-llvm -c -o main.bc  main.cpp
llc.exe -march=arm -mcpu=cortex-a57 -mattr=a57,armv8-a,v8 -meabi=gnu -o main.s main.bc
SCP main.s to the arm machine. SSH and
gcc main.s (using gcc as a test. LLVM should do this.)

Error

main.s: Assembler messages:
main.s:2: Error: unknown pseudo-op: `.syntax'
main.s:3: Error: unknown pseudo-op: `.eabi_attribute'
main.s:9: Error: unknown pseudo-op: `.fpu'
main.s:26: Error: junk at end of line, first unrecognized character is `@'
main.s:29: Error: unknown pseudo-op: `.code'
main.s:31: Error: unknown pseudo-op: `.fnstart'
main.s:32: Error: junk at end of line, first unrecognized character is `@'
main.s:34: Error: operand 1 should be an integer register -- `mov r2,#0'
main.s:41: Error: operand 1 should be an integer or stack pointer register -- `add r0,r0,#1'
main.s:45: Error: unknown mnemonic `bx' -- `bx lr'
main.s:48: Error: unknown pseudo-op: `.cantunwind'
main.s:49: Error: unknown pseudo-op: `.fnend'
main.s:50: Error: junk at end of line, first unrecognized character is `@'

So I tried to target only Windows

clang.exe  -emit-llvm -c -o main.bc  main.cpp
llc.exe -march=x86 -c -o main.s main.bc
ld.lld.exe main.s

Error

ld.lld.exe: error: main.s:1: unknown directive: .text

Then, instead of ld.lld.exe use gcc (Again using gcc as a test. LLVM should do this.)

clang.exe  -emit-llvm -c -o main.bc  main.cpp
llc.exe -march=x86 -c -o main.s main.bc
gcc main.s -o main.exe

That works. To test I type

main.exe
echo Exit Code is %errorlevel%

Which returns 42

General question

What are the steps to compile a C++ program under Windows targeting an arm CPU using only LLVM (no gcc, nothing downloaded from ARM)?

Specific questions

1. Can the tools that come with self-compiled LLVM (e.g. clang.exe, llc.exe, lld.exe) compile an executable on Windows targeting arm? E.g is lld still under development?
2. Why does my attempt to compile and link under Windows, targeting Windows fail?
3. Where do the header files and libraries (e.g. libstdc++) come from when linking on the host for the target? I suppose I need to get those from the arm machine? Copy them to the host and tell the linker where to find them ? Is that correct?

---

Update

So I originally tried Cross-compilation using Clang

clang.exe --target=arm --sysroot=c:\code\clang\FromCmdLine main.cpp  -v

The result is

clang.exe: error: linker (via gcc) command failed with exit code 1 (use -v to see invocation)

And the details of -v are

 "C:\\llvm\\clang.exe" -cc1 -triple armv4t-- -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -discard-value-names -main-file-name main.cpp -mrelocation-model static -mthread-model posix -mdisable-fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -target-cpu arm7tdmi -target-feature +soft-float -target-feature +soft-float-abi -target-feature -fp-only-sp -target-feature -d16 -target-feature -vfp2 -target-feature -vfp3 -target-feature -fp16 -target-feature -vfp4 -target-feature -fp-armv8 -target-feature -neon -target-feature -crypto -target-feature +strict-align -target-abi aapcs -msoft-float -mfloat-abi soft -fallow-half-arguments-and-returns -dwarf-column-info -debugger-tuning=gdb -v -resource-dir "c:\\llvm\\clang\\7.0.0" -isysroot "c:\\code" -fdeprecated-macro -fdebug-compilation-dir "c:\\code" -ferror-limit 19 -fmessage-length 293 -fno-signed-char -fobjc-runtime=gcc -fcxx-exceptions -fexceptions -fdiagnostics-show-option -fcolor-diagnostics -o "C:\\Users\\AppData\\Local\\Temp\\main-b17d06.o" -x c++ main.cpp
clang -cc1 version 7.0.0 based upon LLVM 7.0.0svn default target x86_64-pc-win32
ignoring nonexistent directory "c:\code\usr/local/include"
ignoring nonexistent directory "c:\code\usr/include"
#include "..." search starts here:
#include <...> search starts here:
 C:\llvm\clang\7.0.0\include
End of search list.
 "C:\\MinGW\\bin\\gcc.exe" "--sysroot=c:\\code" -v -o a.out "C:\\Users\\AppData\\Local\\Temp\\main-b17d06.o"
Using built-in specs.
COLLECT_GCC=C:\MinGW\bin\gcc.exe
COLLECT_LTO_WRAPPER=c:/mingw/bin/../libexec/gcc/mingw32/6.3.0/lto-wrapper.exe
Target: mingw32
Configured with: ../src/gcc-6.3.0/configure --build=x86_64-pc-linux-gnu --host=mingw32 --target=mingw32 --with-gmp=/mingw --with-mpfr --with-mpc=/mingw --with-isl=/mingw --prefix=/mingw --disable-win32-registry --with-arch=i586 --with-tune=generic --enable-languages=c,c++,objc,obj-c++,fortran,ada --with-pkgversion='MinGW.org GCC-6.3.0-1' --enable-static --enable-shared --enable-threads --with-dwarf2 --disable-sjlj-exceptions --enable-version-specific-runtime-libs --with-libiconv-prefix=/mingw --with-libintl-prefix=/mingw --enable-libstdcxx-debug --enable-libgomp --disable-libvtv --enable-nls
Thread model: win32
gcc version 6.3.0 (MinGW.org GCC-6.3.0-1)
COMPILER_PATH=c:/mingw/bin/../libexec/gcc/mingw32/6.3.0/;c:/mingw/bin/../libexec/gcc/;c:/mingw/bin/../lib/gcc/mingw32/6.3.0/../../../../mingw32/bin/
LIBRARY_PATH=c:/mingw/bin/../lib/gcc/mingw32/6.3.0/;c:/mingw/bin/../lib/gcc/;c:/mingw/bin/../lib/gcc/mingw32/6.3.0/../../../../mingw32/lib/;c:/mingw/bin/../lib/gcc/mingw32/6.3.0/../../../;c:/code/clang/FromCmdLine/lib/
COLLECT_GCC_OPTIONS='-v' '-o' 'a.out' '-mtune=generic' '-march=i586'
 c:/mingw/bin/../libexec/gcc/mingw32/6.3.0/collect2.exe -plugin c:/mingw/bin/../libexec/gcc/mingw32/6.3.0/liblto_plugin-0.dll -plugin-opt=c:/mingw/bin/../libexec/gcc/mingw32/6.3.0/lto-wrapper.exe -plugin-opt=-fresolution=C:\Users\AppData\Local\Temp\ccufvVIA.res -plugin-opt=-pass-through=-lmingw32 -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_eh -plugin-opt=-pass-through=-lmoldname -plugin-opt=-pass-through=-lmingwex -plugin-opt=-pass-through=-lmsvcrt -plugin-opt=-pass-through=-ladvapi32 -plugin-opt=-pass-through=-lshell32 -plugin-opt=-pass-through=-luser32 -plugin-opt=-pass-through=-lkernel32 -plugin-opt=-pass-through=-lmingw32 -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_eh -plugin-opt=-pass-through=-lmoldname -plugin-opt=-pass-through=-lmingwex -plugin-opt=-pass-through=-lmsvcrt --sysroot=c:\code\clang\FromCmdLine -Bdynamic -o a.out c:/mingw/bin/../lib/gcc/mingw32/6.3.0/../../../crt2.o c:/mingw/bin/../lib/gcc/mingw32/6.3.0/crtbegin.o -Lc:/mingw/bin/../lib/gcc/mingw32/6.3.0 -Lc:/mingw/bin/../lib/gcc -Lc:/mingw/bin/../lib/gcc/mingw32/6.3.0/../../../../mingw32/lib -Lc:/mingw/bin/../lib/gcc/mingw32/6.3.0/../../.. -Lc:/code/clang/FromCmdLine/lib C:\Users\AppData\Local\Temp\main-b17d06.o -lmingw32 -lgcc -lgcc_eh -lmoldname -lmingwex -lmsvcrt -ladvapi32 -lshell32 -luser32 -lkernel32 -lmingw32 -lgcc -lgcc_eh -lmoldname -lmingwex -lmsvcrt c:/mingw/bin/../lib/gcc/mingw32/6.3.0/crtend.o
c:/mingw/bin/../lib/gcc/mingw32/6.3.0/../../../../mingw32/bin/ld.exe: C:\Users\AppData\Local\Temp\main-b17d06.o: Relocations in generic ELF (EM: 40)
C:\Users\AppData\Local\Temp\main-b17d06.o: error adding symbols: File in wrong format

---

Update

This does not fully answer my question but it does help me to progress.

For a better understanding I found crosstool-NG useful, especially their documentation (chapters 1 to 5).

Then I read the cmake cross compiling documentation.

The I wrote a small cmake C++ test.

Helloworld.cpp

#include <iostream>
int main(int argc, char *argv[])
{
   std::cout << "Hello World!" << std::endl;
   return 0;
}

CMakeLists.txt

cmake_minimum_required(VERSION 2.8.9)
project (hello)
add_executable(hello helloworld.cpp)

Target specific configuration for cmake. This is from 4.

set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR arm)

set(CMAKE_SYSROOT /home/user/x-tools/aarch64-unknown-linux-gnueabi/aarch64-unknown-linux-gnueabi/sysroot/)
set(CMAKE_STAGING_PREFIX /home/user/crosscompile/stage)

set(tools /home/user/x-tools/aarch64-unknown-linux-gnueabi)
set(CMAKE_C_COMPILER ${tools}/bin/aarch64-unknown-linux-gnueabi-gcc)
set(CMAKE_CXX_COMPILER ${tools}/bin/aarch64-unknown-linux-gnueabi-g++)

set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)

And the command line

cmake -DCMAKE_TOOLCHAIN_FILE=../toolchain_file.txt ..

That cross compiles to ARM and the program runs on the ARM machine.

But this does not use LLVM / Clang. To use LLVM I thought of changing the toolchain configuration to use

set(tools /usr/bin)
set(CMAKE_C_COMPILER ${tools}/clang)
set(CMAKE_CXX_COMPILER ${tools}/clang++)

That failed because that bin folder is for the host machine.

I also tried using the AArch64 download from http://releases.llvm.org/download.html. Yes that also did not work.

So in summary this what is required.

1. A sysroot folder with the lib and include folders for the target system. Okay there needs to be more in that sysroot folder than lib and include.
2. A toolchain (compiler, assembler, linker) for the target system.

Answer 1

This is what I had to do, to get a proof-of-concept working, to cross compile using only llvm, with host=linux x86_64 and target = DrivePX (arm aarch64). (Also worked with host=Windows 10 x86_64.)

I recommend a tool like croostool-ng which sets up a cross compilation toolchain for you, but the steps below show what's going on behind the scenes, and it uses only llvm.

1. On the host computer, checkout and compile LLVM including Clang
   • see http://llvm.org/docs/GettingStarted.html.
   • This takes a long time so do a release build. Also make sure you have plenty of space (GB).
   • To tell cmake to do a release build e.g.
   • CC=gcc CXX=g++ cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release ../llvm/
   • Then call make.
2. Call clang on the host computer.
   • Remember to set the target, e.g. --target=aarch64-linux-gnu
   • Tell clang to use the llvm linker, i.e. -fuse-ld=lld
   • clang --target=aarch64-linux-gnu -v main.cpp -o main -fuse-ld=lld
3. There will be a lot of linker errors

ld.lld: error: unable to find library -lgcc

ld.lld: error: unable to find library -lgcc_s

ld.lld: error: unable to find library -lc

4. And some more errors about missing crtX.o files, e.g.

ld.lld: error: cannot open crt1.o: No such file or directory

5. On the target machine find, all the libgcc, libc, etc. files.
   • Remember to copy the fully versioned libraries, e.g. for libgcc_s.so copy libgcc_s.so.1
   • Copy these files to the host computer.
   • Tell clang where to find these libraries, e.g. Copy the files into a folder called libs
   • clang --target=aarch64-linux-gnu -v main.cpp -o main -fuse-ld=lld -L./libs
   • Also remember that the linker is looking for e.g. libgcc_s.so so create symbolic links to e.g. libgcc_s.so.1.
6. Copy the crtX.o files from the target to the host. Put them next to the main.cpp.
7. Then one more error

ld.lld: error: undefined symbol: __libc_csu_fini

• That symbol is located in libc_nonshared.a. And this answer helped.

libc.so is a linker script which tells the linker to pull in the shared library libc.so.6, and a non-shared portion, libc_nonshared.a

• On the host find and open libc.so and see where libc.so.6 and libc_nonshared.a are located.
• Copy them to host.
• Tell the linker to use those 2 libraries, i.e. -lc -lc_nonshared
• clang --target=aarch64-linux-gnu -v main.cpp -o main -fuse-ld=lld -L./libs -lc -lc_nonshared

My short text code only depends on libc, libgcc and requires no header files. If your code requires other libraries and header files you would have to copy them from the target to the host.

---

Update Read this question if you are wondering about libgcc.