Yocto Warrior Bitbake Recipe for PyTorch for NVIDIA Jetson Nano

Question

I am trying to create a simple Yocto Python Recipe for Python 3 PyTorch. The target is an SD Card Image for the NVIDIA Jetson Nano produced by Yocto from the meta-tegra layer. I can successfully compile and boot an image from meta-tegra without these recipes.

NVIDIA themselves have compiled and released the ".whl" Python packages and they are found here: https://devtalk.nvidia.com/default/topic/1048776/official-tensorflow-for-jetson-nano-/

Using their build instructions, I tried to write a Bitbake recipe to install PyTorch as follows:

SUMMARY = "Facebook PyTorch AI"
DESCRIPTION = "Facebook PyTorch AI"
HOMEPAGE = "https://pytorch.org/"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://LICENSE;md5=acf4d595f99e159bf31797aa872aef57"

inherit pypi
inherit setuptools3

#BBCLASSEXTEND = "native nativesdk"

DEPENDS += "python3-pytest-runner-native python3-pyyaml-native cmake-native"

do_configure_prepend() {
  USE_NCCL=0
  USE_DISTRIBUTED=0
  TORCH_CUDA_ARCH_LIST="5.3;6.2;7.2"
}

do_compile_prepend() {
  USE_NCCL=0
  USE_DISTRIBUTED=0
  TORCH_CUDA_ARCH_LIST="5.3;6.2;7.2"
}

SRC_URI = "gitsm://github.com/pytorch/pytorch.git;protocol=https"
SRCREV = "a3346e100e7f4e7ec90f18b7befcccc47d5a1c82"

S = "${WORKDIR}/git"

The problem is that I hit the following error. I think it has to do with my environment variables not being picked up becuase the error includes "-DUSE_CUDA=False", however when I look at the bitbake environment ( bitbake -e ), it seems to be there.

Here is the error:

**NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
ERROR: python3-torch-1.1.0-r0 do_compile: 'python3 setup.py build ' execution failed.
ERROR: python3-torch-1.1.0-r0 do_compile: Function failed: do_compile (log file is located at /home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/temp/log.do_compile.27625)
ERROR: Logfile of failure stored in: /home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/temp/log.do_compile.27625
Log data follows:
| DEBUG: Executing shell function do_compile
| -- The CXX compiler identification is GNU 7.2.1
| -- The C compiler identification is GNU 7.2.1
| -- Check for working CXX compiler: /home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/bin/aarch64-poky-linux/aarch64-poky-linux-g++
| -- Check for working CXX compiler: /home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/bin/aarch64-poky-linux/aarch64-poky-linux-g++ -- works
| -- Detecting CXX compiler ABI info
| -- Detecting CXX compiler ABI info - done
| -- Detecting CXX compile features
| -- Detecting CXX compile features - done
| -- Check for working C compiler: /home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/bin/aarch64-poky-linux/aarch64-poky-linux-gcc
| -- Check for working C compiler: /home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/bin/aarch64-poky-linux/aarch64-poky-linux-gcc -- works
| -- Detecting C compiler ABI info
| -- Detecting C compiler ABI info - done
| -- Detecting C compile features
| -- Detecting C compile features - done
| -- Not forcing any particular BLAS to be found
| -- Performing Test COMPILER_WORKS
| -- Performing Test COMPILER_WORKS - Failed
| CMake Error at cmake/MiscCheck.cmake:52 (message):
|   Could not run a simple program built with your compiler.  If you are trying
|   to use -fsanitize=address, make sure libasan is properly installed on your
|   system (you can confirm if the problem is this by attempting to build and
|   run a small program.)
| Call Stack (most recent call first):
|   CMakeLists.txt:271 (include)
| 
| 
| -- Configuring incomplete, errors occurred!
| See also "/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/git/build/CMakeFiles/CMakeOutput.log".
| See also "/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/git/build/CMakeFiles/CMakeError.log".
| Building wheel torch-1.2.0a0+a3346e1
| -- Building version 1.2.0a0+a3346e1
| cmake -DBUILD_AR=ar -DBUILD_AS=as  -DBUILD_CC=gcc  -DBUILD_CCLD=gcc  -DBUILD_CFLAGS=-isystem/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/include -O2 -pipe -DBUILD_CPP=gcc  -E -DBUILD_CPPFLAGS=-isystem/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/include -DBUILD_CXX=g++  -DBUILD_CXXFLAGS=-isystem/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/include -O2 -pipe -DBUILD_FC=gfortran  -DBUILD_LD=ld  -DBUILD_LDFLAGS=-L/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/lib                         -L/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/lib                         -Wl,--enable-new-dtags                         -Wl,-rpath-link,/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/lib                         -Wl,-rpath-link,/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/lib                         -Wl,-rpath,/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/lib                         -Wl,-rpath,/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/lib                         -Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2 -DBUILD_NM=nm -DBUILD_PYTHON=True -DBUILD_RANLIB=ranlib -DBUILD_STRIP=strip -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS= -O2 -pipe -g -feliminate-unused-debug-types -fdebug-prefix-map=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0=/usr/src/debug/python3-torch/1.1.0-r0                      -fdebug-prefix-map=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot=                      -fdebug-prefix-map=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native=   -DCMAKE_C_FLAGS= -O2 -pipe -g -feliminate-unused-debug-types -fdebug-prefix-map=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0=/usr/src/debug/python3-torch/1.1.0-r0                      -fdebug-prefix-map=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot=                      -fdebug-prefix-map=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native=   -DCMAKE_EXE_LINKER_FLAGS=-Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -fstack-protector-strong -Wl,-z,relro,-z,now -DCMAKE_INSTALL_PREFIX=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/git/torch -DCMAKE_PREFIX_PATH=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot/usr/lib/python3.7/site-packages -DCMAKE_SHARED_LINKER_FLAGS=-Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -fstack-protector-strong -Wl,-z,relro,-z,now -DINSTALL_TEST=True -DPYTHON_EXECUTABLE=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/bin/python3-native/python3 -DPYTHON_INCLUDE_DIR=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot/usr/include/python3.7m -DPYTHON_LIBRARY=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot/usr/lib/libpython3.7m.so.1.0 -DTORCH_BUILD_VERSION=1.2.0a0+a3346e1 -DUSE_CUDA=False -DUSE_DISTRIBUTED=True -DUSE_FBGEMM=True -DUSE_NCCL=False -DUSE_NUMPY=True -DUSE_QNNPACK=True -DUSE_SYSTEM_EIGEN_INSTALL=OFF -DUSE_SYSTEM_NCCL=False /home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/git
| Traceback (most recent call last):
|   File "setup.py", line 749, in <module>
|     build_deps()
|   File "setup.py", line 323, in build_deps
|     cmake=cmake)
|   File "/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/git/tools/build_pytorch_libs.py", line 61, in build_caffe2
|     rerun_cmake)
|   File "/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/git/tools/setup_helpers/cmake.py", line 318, in generate
|     self.run(args, env=my_env)
|   File "/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/git/tools/setup_helpers/cmake.py", line 93, in run
|     check_call(command, cwd=self.build_dir, env=env)
|   File "/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/lib/python3.7/subprocess.py", line 347, in check_call
|     raise CalledProcessError(retcode, cmd)
| subprocess.CalledProcessError: Command '['cmake', '-DBUILD_AR=ar', '-DBUILD_AS=as ', '-DBUILD_CC=gcc ', '-DBUILD_CCLD=gcc ', '-DBUILD_CFLAGS=-isystem/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/include -O2 -pipe', '-DBUILD_CPP=gcc  -E', '-DBUILD_CPPFLAGS=-isystem/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/include', '-DBUILD_CXX=g++ ', '-DBUILD_CXXFLAGS=-isystem/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/include -O2 -pipe', '-DBUILD_FC=gfortran ', '-DBUILD_LD=ld ', '-DBUILD_LDFLAGS=-L/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/lib                         -L/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/lib                         -Wl,--enable-new-dtags                         -Wl,-rpath-link,/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/lib                         -Wl,-rpath-link,/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/lib                         -Wl,-rpath,/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/lib                         -Wl,-rpath,/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/lib                         -Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2', '-DBUILD_NM=nm', '-DBUILD_PYTHON=True', '-DBUILD_RANLIB=ranlib', '-DBUILD_STRIP=strip', '-DBUILD_TEST=True', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_CXX_FLAGS= -O2 -pipe -g -feliminate-unused-debug-types -fdebug-prefix-map=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0=/usr/src/debug/python3-torch/1.1.0-r0                      -fdebug-prefix-map=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot=                      -fdebug-prefix-map=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native=  ', '-DCMAKE_C_FLAGS= -O2 -pipe -g -feliminate-unused-debug-types -fdebug-prefix-map=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0=/usr/src/debug/python3-torch/1.1.0-r0                      -fdebug-prefix-map=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot=                      -fdebug-prefix-map=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native=  ', '-DCMAKE_EXE_LINKER_FLAGS=-Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -fstack-protector-strong -Wl,-z,relro,-z,now', '-DCMAKE_INSTALL_PREFIX=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/git/torch', '-DCMAKE_PREFIX_PATH=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot/usr/lib/python3.7/site-packages', '-DCMAKE_SHARED_LINKER_FLAGS=-Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -fstack-protector-strong -Wl,-z,relro,-z,now', '-DINSTALL_TEST=True', '-DPYTHON_EXECUTABLE=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot-native/usr/bin/python3-native/python3', '-DPYTHON_INCLUDE_DIR=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot/usr/include/python3.7m', '-DPYTHON_LIBRARY=/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/recipe-sysroot/usr/lib/libpython3.7m.so.1.0', '-DTORCH_BUILD_VERSION=1.2.0a0+a3346e1', '-DUSE_CUDA=False', '-DUSE_DISTRIBUTED=True', '-DUSE_FBGEMM=True', '-DUSE_NCCL=False', '-DUSE_NUMPY=True', '-DUSE_QNNPACK=True', '-DUSE_SYSTEM_EIGEN_INSTALL=OFF', '-DUSE_SYSTEM_NCCL=False', '/home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/git']' returned non-zero exit status 1.
| ERROR: 'python3 setup.py build ' execution failed.
| WARNING: exit code 1 from a shell command.
| ERROR: Function failed: do_compile (log file is located at /home/ubuntu/Desktop/comp-jetson-yocto/build/tmp/work/aarch64-poky-linux/python3-torch/1.1.0-r0/temp/log.do_compile.27625)
ERROR: Task (/home/ubuntu/Desktop/comp-jetson-yocto/layers/meta-comp/recipes-python/python3-torch/python3-torch_1.1.0.bb:do_compile) failed with exit code '1'
NOTE: Tasks Summary: Attempted 1770 tasks of which 1763 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  /home/ubuntu/Desktop/comp-jetson-yocto/layers/meta-comp/recipes-python/python3-torch/python3-torch_1.1.0.bb:do_compile
Summary: There was 1 WARNING message shown.
Summary: There were 2 ERROR messages shown, returning a non-zero exit code.**

Answer 1

root-cause of the problem is

if (NOT INTERN_BUILD_MOBILE)
  # ---[ Check that our programs run.  This is different from the native CMake
  # compiler check, which just tests if the program compiles and links.  This is
  # important because with ASAN you might need to help the compiled library find
  # some dynamic libraries.
  cmake_push_check_state(RESET)
  CHECK_C_SOURCE_RUNS("
  int main() { return 0; }
  " COMPILER_WORKS)
  if (NOT COMPILER_WORKS)
    # Force cmake to retest next time around
    unset(COMPILER_WORKS CACHE)
    message(FATAL_ERROR
        "Could not run a simple program built with your compiler. "
        "If you are trying to use -fsanitize=address, make sure "
        "libasan is properly installed on your system (you can confirm "
        "if the problem is this by attempting to build and run a "
        "small program.)")
  endif()
  cmake_pop_check_state()
endif()

this is running program compiled. but we are cross compiling here. can you try setting INTERN_BUILD_MOBILE

set(INTERN_BUILD_MOBILE ON)

I don't have direct source which will build on jetson-nano but following is close may help progressing. you can refer to this pull request for cross-compiling for android.

Happy to answer anything else you may need.