Cannot build from source on NVIDIA Jetson Xavier NX

Question

Cannot build from source on NVIDIA Jetson Xavier NX

markub3327 opened this issue 2 years ago · comments

Martin Kubovčík commented 2 years ago

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04.5 LTS
TensorFlow version and how it was installed (source or binary): TF 2.9.1, binary from NVIDIA repo (https://developer.download.nvidia.com/compute/redist/jp/v50/tensorflow/)
TensorFlow-Addons version and how it was installed (source or binary): v0.17.1, source (ZIP from https://github.com/tensorflow/addons/releases/tag/v0.17.1)
Python version: 3.8.10
Is GPU used? (yes/no): yes
Bazel version: 3.7.2 arm64
GCC version: 9.4.0

Describe the bug

The build process crash with error:

martin@NVIDIA:~/Desktop/addons-0.17.1$ export TF_CUDA_VERSION="11.4"
martin@NVIDIA:~/Desktop/addons-0.17.1$ export CUDNN_INSTALL_PATH="/usr/lib/aarch64-linux-gnu"
martin@NVIDIA:~/Desktop/addons-0.17.1$ export CUDA_TOOLKIT_PATH="/usr/local/cuda"

martin@NVIDIA:~/Desktop/addons-0.17.1$ python3 ./configure.py

Configuring TensorFlow Addons to be built from source...
> Building GPU & CPU ops

Build configurations successfully written to .bazelrc :

build --action_env TF_HEADER_DIR="/usr/local/lib/python3.8/dist-packages/tensorflow/include"
build --action_env TF_SHARED_LIBRARY_DIR="/usr/local/lib/python3.8/dist-packages/tensorflow/python"
build --action_env TF_SHARED_LIBRARY_NAME="_pywrap_tensorflow_internal.so"
build --action_env TF_CXX11_ABI_FLAG="1"
build --spawn_strategy=standalone
build --strategy=Genrule=standalone
build  --experimental_repo_remote_exec
build -c opt
build --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=1"
build --cxxopt=-std=c++14
build --host_cxxopt=-std=c++14
build --action_env TF_NEED_CUDA="1"
build --action_env CUDA_TOOLKIT_PATH="/usr/local/cuda"
build --action_env CUDNN_INSTALL_PATH="/usr/lib/aarch64-linux-gnu"
build --action_env TF_CUDA_VERSION="11.4"
build --action_env TF_CUDNN_VERSION="8"
test --config=cuda
build --config=cuda
build:cuda --define=using_cuda=true --define=using_cuda_nvcc=true
build:cuda --crosstool_top=@ubuntu20.04-gcc9_manylinux2014-cuda11.2-cudnn8.1-tensorrt7.2_config_cuda//crosstool:toolchain

martin@NVIDIA:~/Desktop/addons-0.17.1$ 
martin@NVIDIA:~/Desktop/addons-0.17.1$ bazel build build_pip_pkg
Starting local Bazel server and connecting to it...
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/tensorflow/runtime/archive/093ed77f7d50f75b376f40a71ea86e08cedb8b80.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
DEBUG: /home/martin/.cache/bazel/_bazel_martin/cef3e79225ec38d81c24e1a129f67826/external/org_tensorflow/third_party/repo.bzl:124:14: 
Warning: skipping import of repository 'cub_archive' because it already exists.
WARNING: Download from https://mirror.bazel.build/github.com/bazelbuild/rules_cc/archive/081771d4a0e9d7d3aa0eed2ef389fa4700dfb23e.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
DEBUG: Rule 'io_bazel_rules_docker' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1596824487 -0400"
DEBUG: Repository io_bazel_rules_docker instantiated at:
  /home/martin/Desktop/addons-0.17.1/WORKSPACE:45:14: in <toplevel>
  /home/martin/.cache/bazel/_bazel_martin/cef3e79225ec38d81c24e1a129f67826/external/org_tensorflow/tensorflow/workspace0.bzl:107:34: in workspace
  /home/martin/.cache/bazel/_bazel_martin/cef3e79225ec38d81c24e1a129f67826/external/bazel_toolchains/repositories/repositories.bzl:35:23: in repositories
Repository rule git_repository defined at:
  /home/martin/.cache/bazel/_bazel_martin/cef3e79225ec38d81c24e1a129f67826/external/bazel_tools/tools/build_defs/repo/git.bzl:199:33: in <toplevel>
DEBUG: /home/martin/.cache/bazel/_bazel_martin/cef3e79225ec38d81c24e1a129f67826/external/bazel_tools/tools/cpp/lib_cc_configure.bzl:118:10: 
Auto-Configuration Warning: 'TMP' environment variable is not set, using 'C:\Windows\Temp' as default
WARNING: /home/martin/.cache/bazel/_bazel_martin/cef3e79225ec38d81c24e1a129f67826/external/local_config_tf/BUILD:10951:8: target '_pywrap_tensorflow_internal.so' is both a rule and a file; please choose another name for the rule
INFO: Analyzed target //:build_pip_pkg (47 packages loaded, 570 targets configured).
INFO: Found 1 target...
ERROR: /home/martin/Desktop/addons-0.17.1/BUILD:1:10: failed to create symbolic link 'build_pip_pkg': file 'build_deps/build_pip_pkg.sh' is not executable
Target //:build_pip_pkg failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 429.366s, Critical Path: 0.94s
INFO: 97 processes: 96 internal, 1 local.
FAILED: Build did NOT complete successfully
martin@NVIDIA:~/Desktop/addons-0.17.1$ chmod +x build_deps/build_pip_pkg.sh 
martin@NVIDIA:~/Desktop/addons-0.17.1$ bazel build build_pip_pkg
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/tensorflow/runtime/archive/093ed77f7d50f75b376f40a71ea86e08cedb8b80.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
DEBUG: /home/martin/.cache/bazel/_bazel_martin/cef3e79225ec38d81c24e1a129f67826/external/org_tensorflow/third_party/repo.bzl:124:14: 
Warning: skipping import of repository 'cub_archive' because it already exists.
DEBUG: /home/martin/.cache/bazel/_bazel_martin/cef3e79225ec38d81c24e1a129f67826/external/bazel_tools/tools/cpp/lib_cc_configure.bzl:118:10: 
Auto-Configuration Warning: 'TMP' environment variable is not set, using 'C:\Windows\Temp' as default
DEBUG: Rule 'io_bazel_rules_docker' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1596824487 -0400"
DEBUG: Repository io_bazel_rules_docker instantiated at:
  /home/martin/Desktop/addons-0.17.1/WORKSPACE:45:14: in <toplevel>
  /home/martin/.cache/bazel/_bazel_martin/cef3e79225ec38d81c24e1a129f67826/external/org_tensorflow/tensorflow/workspace0.bzl:107:34: in workspace
  /home/martin/.cache/bazel/_bazel_martin/cef3e79225ec38d81c24e1a129f67826/external/bazel_toolchains/repositories/repositories.bzl:35:23: in repositories
Repository rule git_repository defined at:
  /home/martin/.cache/bazel/_bazel_martin/cef3e79225ec38d81c24e1a129f67826/external/bazel_tools/tools/build_defs/repo/git.bzl:199:33: in <toplevel>
WARNING: Download from https://mirror.bazel.build/github.com/bazelbuild/rules_cc/archive/081771d4a0e9d7d3aa0eed2ef389fa4700dfb23e.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
WARNING: /home/martin/.cache/bazel/_bazel_martin/cef3e79225ec38d81c24e1a129f67826/external/local_config_tf/BUILD:10951:8: target '_pywrap_tensorflow_internal.so' is both a rule and a file; please choose another name for the rule
INFO: Analyzed target //:build_pip_pkg (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /home/martin/Desktop/addons-0.17.1/tensorflow_addons/custom_ops/seq2seq/BUILD:7:18: C++ compilation of rule '//tensorflow_addons/custom_ops/seq2seq:_beam_search_ops_gpu' failed (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command external/ubuntu20.04-gcc9_manylinux2014-cuda11.2-cudnn8.1-tensorrt7.2_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF ... (remaining 59 argument(s) skipped)
/dt9/usr/bin/gcc: No such file or directory
nvcc fatal   : Failed to preprocess host compiler properties.
Target //:build_pip_pkg failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 64.061s, Critical Path: 62.38s
INFO: 1374 processes: 1370 internal, 4 local.
FAILED: Build did NOT complete successfully

pkuppusamy77 · Answer 1 · Thu Feb 02 2023 19:33:08 GMT+0800 (China Standard Time)

Hi I have a same issue when i try to build tensorflow-addons in Jetson Xavier NX

Did you get any Solution
Thanks