Build failure on Linux at current head
mberaha opened this issue · comments
Building the jaxlib produces the following trace
python3 build/build.py
_ _ __ __
| | / \ \ \/ /
_ | |/ _ \ \ /
| |_| / ___ \/ \
\___/_/ \/_/\_\
Downloading bazel from: https://github.com/bazelbuild/bazel/releases/download/5.0.0/bazel-5.0.0-linux-x86_64
bazel-5.0.0-linux-x86_64 [########################################] 100%
Bazel binary path: ./bazel-5.0.0-linux-x86_64
Bazel version: 5.0.0
Python binary path: /usr/bin/python3
Python version: 3.8
NumPy version: 1.21.2
MKL-DNN enabled: yes
Target CPU: x86_64
Target CPU features: release
CUDA enabled: no
TPU enabled: no
ROCm enabled: no
Building XLA and installing it in the jaxlib source tree...
./bazel-5.0.0-linux-x86_64 run --verbose_failures=true --config=avx_posix --config=mkl_open_source_only :build_wheel -- --output_path=/home/beraha/usr/jax/dist --cpu=x86_64
Starting local Bazel server and connecting to it...
INFO: Options provided by the client:
Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'run' from /home/beraha/usr/jax/.bazelrc:
Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'run' from /home/beraha/usr/jax/.bazelrc:
Inherited 'build' options: --apple_platform_type=macos --macos_minimum_os=10.9 --announce_rc --define open_source_build=true --spawn_strategy=standalone --enable_platform_specific_config --define=no_aws
_support=true --define=no_gcp_support=true --define=no_hdfs_support=true --define=no_kafka_support=true --define=no_ignite_support=true --define=grpc_no_ares=true -c opt --config=short_logs --copt=-DMLIR_
PYTHON_PACKAGE_PREFIX=jaxlib.mlir.
INFO: Reading rc options for 'run' from /home/beraha/usr/jax/.jax_configure.bazelrc:
Inherited 'build' options: --strategy=Genrule=standalone --repo_env PYTHON_BIN_PATH=/usr/bin/python3 --action_env=PYENV_ROOT --python_path=/usr/bin/python3 --distinct_host_configuration=false
INFO: Found applicable config definition build:short_logs in file /home/beraha/usr/jax/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:avx_posix in file /home/beraha/usr/jax/.bazelrc: --copt=-mavx --host_copt=-mavx
INFO: Found applicable config definition build:mkl_open_source_only in file /home/beraha/usr/jax/.bazelrc: --define=tensorflow_mkldnn_contraction_kernel=1
INFO: Found applicable config definition build:linux in file /home/beraha/usr/jax/.bazelrc: --config=posix --copt=-Wno-stringop-truncation --copt=-Wno-array-parameter
INFO: Found applicable config definition build:posix in file /home/beraha/usr/jax/.bazelrc: --copt=-fvisibility=hidden --copt=-Wno-sign-compare --cxxopt=-std=c++14 --host_cxxopt=-std=c++14
Loading:
Loading: 0 packages loaded
Loading: 0 packages loaded
Analyzing: target //build:build_wheel (1 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (38 packages loaded, 53 targets configured)
Analyzing: target //build:build_wheel (134 packages loaded, 879 targets configured)
Analyzing: target //build:build_wheel (200 packages loaded, 8716 targets configured)
Analyzing: target //build:build_wheel (208 packages loaded, 14679 targets configured)
Analyzing: target //build:build_wheel (209 packages loaded, 14718 targets configured)
INFO: Analyzed target //build:build_wheel (220 packages loaded, 15453 targets configured).
INFO: Found 1 target...
[0 / 27] [Prepa] BazelWorkspaceStatusAction stable-status.txt
[100 / 2,620] Compiling llvm/lib/Demangle/ItaniumDemangle.cpp; 0s local ... (20 actions, 19 running)
[119 / 2,620] Compiling llvm/lib/Demangle/ItaniumDemangle.cpp; 2s local ... (20 actions, 19 running)
[143 / 2,620] Compiling llvm/lib/Support/ScopedPrinter.cpp; 2s local ... (20 actions, 19 running)
[174 / 2,620] Compiling llvm/lib/Support/APFloat.cpp; 3s local ... (20 actions, 19 running)
[201 / 2,620] Compiling llvm/lib/Support/TimeProfiler.cpp; 2s local ... (20 actions, 19 running)
[231 / 2,620] Compiling llvm/lib/Support/CommandLine.cpp; 5s local ... (19 actions, 18 running)
[268 / 2,626] Compiling llvm/lib/TableGen/TGParser.cpp; 7s local ... (20 actions running)
[285 / 2,626] Compiling llvm/lib/Support/ItaniumManglingCanonicalizer.cpp; 7s local ... (20 actions running)
[333 / 2,647] Compiling mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp; 9s local ... (20 actions running)
ERROR: /home/beraha/.cache/bazel/_bazel_beraha/24259ef897c8798bfdef130aeefe1099/external/llvm-project/mlir/BUILD.bazel:1250:18: TdGenerate external/llvm-project/mlir/include/mlir/Dialect/Async/IR/AsyncOps
.h.inc failed: (Illegal instruction): mlir-tblgen failed: error executing command
(cd /home/beraha/.cache/bazel/_bazel_beraha/24259ef897c8798bfdef130aeefe1099/execroot/__main__ && \
exec env - \
PATH=/home/beraha/.local/bin:/home/beraha/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \
bazel-out/k8-opt-exec-50AE0418/bin/external/llvm-project/mlir/mlir-tblgen -gen-op-decls external/llvm-project/mlir/include/mlir/Dialect/Async/IR/AsyncOps.td -I external/llvm-project/mlir/include -I baze
l-out/k8-opt/bin/external/llvm-project/mlir/include -I external/llvm-project/ -I bazel-out/k8-opt/bin/external/llvm-project/ -I external/llvm-project/mlir/include/mlir/Dialect/Async/IR -I bazel-out/k8-opt
/bin/external/llvm-project/mlir/include/mlir/Dialect/Async/IR -o bazel-out/k8-opt/bin/external/llvm-project/mlir/include/mlir/Dialect/Async/IR/AsyncOps.h.inc)
# Configuration: db954bc994bdf0813e33752680e6cccedb3a7cf231e2f59f48da01e7e4fe37c3
# Execution platform: @local_execution_config_platform//:platform
ERROR: /home/beraha/.cache/bazel/_bazel_beraha/24259ef897c8798bfdef130aeefe1099/external/llvm-project/mlir/BUILD.bazel:1284:18: TdGenerate external/llvm-project/mlir/include/mlir/Dialect/Async/Passes.capi
.cpp.inc failed: (Illegal instruction): mlir-tblgen failed: error executing command
(cd /home/beraha/.cache/bazel/_bazel_beraha/24259ef897c8798bfdef130aeefe1099/execroot/__main__ && \
exec env - \
PATH=/home/beraha/.local/bin:/home/beraha/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \
bazel-out/k8-opt-exec-50AE0418/bin/external/llvm-project/mlir/mlir-tblgen -gen-pass-capi-impl '--prefix=Async' external/llvm-project/mlir/include/mlir/Dialect/Async/Passes.td -I external/llvm-project/ml
ir/include -I bazel-out/k8-opt/bin/external/llvm-project/mlir/include -I external/llvm-project/ -I bazel-out/k8-opt/bin/external/llvm-project/ -I external/llvm-project/mlir/include/mlir/Dialect/Async -I b
azel-out/k8-opt/bin/external/llvm-project/mlir/include/mlir/Dialect/Async -o bazel-out/k8-opt/bin/external/llvm-project/mlir/include/mlir/Dialect/Async/Passes.capi.cpp.inc)
# Configuration: db954bc994bdf0813e33752680e6cccedb3a7cf231e2f59f48da01e7e4fe37c3
# Execution platform: @local_execution_config_platform//:platform
Target //build:build_wheel failed to build
INFO: Elapsed time: 41.083s, Critical Path: 13.80s
INFO: 305 processes: 26 internal, 279 local.
FAILED: Build did NOT complete successfully
ERROR: Build failed. Not running target
FAILED: Build did NOT complete successfully
b''
Traceback (most recent call last):
File "build/build.py", line 527, in <module>
main()
File "build/build.py", line 522, in main
shell(command)
File "build/build.py", line 53, in shell
output = subprocess.check_output(cmd)
File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['./bazel-5.0.0-linux-x86_64', 'run', '--verbose_failures=true', '--config=avx_posix', '--config=mkl_open_source_only', ':build_wheel', '--', '--output_path=/home/b
eraha/usr/jax/dist', '--cpu=x86_64']' returned non-zero exit status 1.
Thanks for the report, we're looking into fixing!
I believe this has been fixed.
Thanks! It is indeed fixed. However, a new error came up. I am compiling on a linux machine without AVX instruction so I'm using the --target_cpu_features="native" flag. Also, the machine does not have a GPU.
python3 build/build.py --target_cpu_features="native"
_ _ __ __
| | / \ \ \/ /
_ | |/ _ \ \ /
| |_| / ___ \/ \
\___/_/ \/_/\_\
Downloading bazel from: https://github.com/bazelbuild/bazel/releases/download/5.1.0/bazel-5.1.0-linux-x86_64
bazel-5.1.0-linux-x86_64 [########################################] 100%
Bazel binary path: ./bazel-5.1.0-linux-x86_64
Bazel version: 5.1.0
Python binary path: /usr/bin/python3
Python version: 3.8
NumPy version: 1.21.2
MKL-DNN enabled: yes
Target CPU: x86_64
Target CPU features: native
CUDA enabled: no
TPU enabled: no
ROCm enabled: no
Building XLA and installing it in the jaxlib source tree...
./bazel-5.1.0-linux-x86_64 run --verbose_failures=true --config=native_arch_posix --config=mkl_open_source_only :build_wheel -- --output_path=/home/beraha/usr/jax/dist --cpu=x86_64
Starting local Bazel server and connecting to it...
... still trying to connect to local Bazel server after 10 seconds ...
... still trying to connect to local Bazel server after 20 seconds ...
INFO: Options provided by the client:
Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'run' from /home/beraha/usr/jax/.bazelrc:
Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'run' from /home/beraha/usr/jax/.bazelrc:
Inherited 'build' options: --apple_platform_type=macos --macos_minimum_os=10.9 --announce_rc --define open_source_build=true --spawn_strategy=standalone --enable_platform_specific_config --experimental_cc_shared_library --define=no_aws_support=true --define=no_gcp_support=true --define=no_hdfs_support=true --define=no_kafka_support=true --define=no_ignite_support=true --define=grpc_no_ares=true -c opt --config=short_logs --copt=-DMLIR_PYTHON_PACKAGE_PREFIX=jaxlib.mlir.
INFO: Reading rc options for 'run' from /home/beraha/usr/jax/.jax_configure.bazelrc:
Inherited 'build' options: --strategy=Genrule=standalone --repo_env PYTHON_BIN_PATH=/usr/bin/python3 --action_env=PYENV_ROOT --python_path=/usr/bin/python3 --distinct_host_configuration=false
INFO: Found applicable config definition build:short_logs in file /home/beraha/usr/jax/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:native_arch_posix in file /home/beraha/usr/jax/.bazelrc: --copt=-march=native --host_copt=-march=native
INFO: Found applicable config definition build:mkl_open_source_only in file /home/beraha/usr/jax/.bazelrc: --define=tensorflow_mkldnn_contraction_kernel=1
INFO: Found applicable config definition build:linux in file /home/beraha/usr/jax/.bazelrc: --config=posix --copt=-Wno-stringop-truncation --copt=-Wno-array-parameter
INFO: Found applicable config definition build:posix in file /home/beraha/usr/jax/.bazelrc: --copt=-fvisibility=hidden --copt=-Wno-sign-compare --cxxopt=-std=c++14 --host_cxxopt=-std=c++14
Loading:
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Analyzing: target //build:build_wheel (1 packages loaded)
Analyzing: target //build:build_wheel (1 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (34 packages loaded, 12 targets configured)
Analyzing: target //build:build_wheel (76 packages loaded, 371 targets configured)
Analyzing: target //build:build_wheel (193 packages loaded, 5966 targets configured)
Analyzing: target //build:build_wheel (206 packages loaded, 14638 targets configured)
Analyzing: target //build:build_wheel (218 packages loaded, 15413 targets configured)
INFO: Analyzed target //build:build_wheel (218 packages loaded, 15413 targets configured).
INFO: Found 1 target...
[13 / 130] [Prepa] BazelWorkspaceStatusAction stable-status.txt
[66 / 127] Compiling absl/synchronization/internal/graphcycles.cc; 1s local ... (20 actions, 19 running)
[154 / 203] Compiling jaxlib/lapack.cc; 2s local ... (20 actions, 19 running)
[166 / 203] Compiling jaxlib/lapack.cc; 5s local ... (20 actions, 19 running)
[207 / 4,514] Compiling src/idl_gen_js_ts.cpp; 8s local ... (20 actions, 19 running)
[239 / 4,514] Compiling src/idl_gen_cpp.cpp; 10s local ... (20 actions, 19 running)
[281 / 4,514] Compiling src/idl_parser.cpp; 13s local ... (20 actions, 19 running)
[319 / 4,514] Compiling src/google/protobuf/descriptor.cc; 6s local ... (20 actions, 19 running)
[382 / 4,516] Compiling src/google/protobuf/compiler/cpp/cpp_message.cc; 3s local ... (4 actions running)
[446 / 5,183] Compiling llvm/lib/TableGen/Record.cpp; 5s local ... (19 actions running)
[538 / 5,183] Compiling mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp; 11s local ... (20 actions running)
[652 / 5,183] Compiling llvm/lib/Support/ItaniumManglingCanonicalizer.cpp; 5s local ... (20 actions, 18 running)
[859 / 5,318] Compiling llvm/lib/TableGen/Record.cpp; 4s local ... (20 actions, 19 running)
[1,208 / 5,447] Compiling llvm/utils/TableGen/CodeGenDAGPatterns.cpp; 14s local ... (16 actions, 15 running)
[1,559 / 5,910] Compiling mlir/tools/mlir-linalg-ods-gen/mlir-linalg-ods-yaml-gen.cpp; 6s local ... (20 actions, 19 running)
[1,811 / 6,093] Compiling mlir/lib/Dialect/Vector/IR/VectorOps.cpp; 10s local ... (20 actions, 19 running)
[2,149 / 6,178] Compiling mlir/lib/Dialect/Vector/IR/VectorOps.cpp; 29s local ... (20 actions, 19 running)
[2,375 / 6,178] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 13s local ... (20 actions, 19 running)
[2,533 / 6,178] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 41s local ... (20 actions, 19 running)
[2,619 / 6,178] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 71s local ... (20 actions, 19 running)
[2,704 / 6,178] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 106s local ... (20 actions, 19 running)
[2,933 / 6,178] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 145s local ... (20 actions, 19 running)
[3,254 / 6,178] Compiling src/cpu/x64/jit_avx512_common_conv_kernel.cpp; 13s local ... (20 actions, 19 running)
[3,546 / 6,178] Compiling tensorflow/compiler/mlir/hlo/lib/Dialect/lhlo/IR/lhlo_ops.cc; 50s local ... (20 actions, 19 running)
[4,034 / 6,501] Compiling mlir/lib/Dialect/Affine/Transforms/SuperVectorize.cpp; 11s local ... (20 actions, 19 running)
[4,279 / 6,540] Compiling mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp; 64s local ... (20 actions, 19 running)
[4,487 / 6,540] Compiling llvm/lib/Analysis/StackSafetyAnalysis.cpp; 10s local ... (20 actions, 19 running)
ERROR: /home/beraha/.cache/bazel/_bazel_beraha/24259ef897c8798bfdef130aeefe1099/external/llvm-project/mlir/BUILD.bazel:3451:11: Compiling mlir/lib/Conversion/GPUCommon/GPUOpsLowering.cpp failed: undeclared inclusion(s) in rule '@llvm-project//mlir:GPUCommonTransforms':
this rule is missing dependency declarations for the following files included by 'mlir/lib/Conversion/GPUCommon/GPUOpsLowering.cpp':
'bazel-out/k8-opt/bin/external/llvm-project/mlir/_virtual_includes/GPUBaseIncGen/mlir/Dialect/GPU/GPUOpsDialect.h.inc'
Target //build:build_wheel failed to build
INFO: Elapsed time: 734.707s, Critical Path: 155.08s
INFO: 4264 processes: 65 internal, 4199 local.
FAILED: Build did NOT complete successfully
ERROR: Build failed. Not running target
FAILED: Build did NOT complete successfully
b''
Traceback (most recent call last):
File "build/build.py", line 527, in <module>
main()
File "build/build.py", line 522, in main
shell(command)
File "build/build.py", line 53, in shell
output = subprocess.check_output(cmd)
File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['./bazel-5.1.0-linux-x86_64', 'run', '--verbose_failures=true', '--config=native_arch_posix', '--config=mkl_open_source_only', ':build_wheel', '--', '--output_path=/home/beraha/usr/jax/dist', '--cpu=x86_64']' returned non-zero exit status 1.
This seems related to a wrong include but for files related to GPU processing.
Hey sorry, I was on a long vacation! Is this still an issue?