Error - bazel run

Question

Error - bazel run

karlie38 opened this issue 2 years ago · comments

Python 3.9.7
OS: Window

Having an error when running bazel to read RIEGELI files.

I followed the readme (install google sdk -> download data using 'gsutil' -> git clone code_contests ->
bazel run -c opt :print_names_and_sources C:\tmp\dm-code_contests\dm-code_contests/code_contests_valid.riegeli
).
I saw same issue with me (#4) and saw the answer of suggesting change of python version (3.x -> 2.x). But, "local variable 'python_include' is referenced before assignment" error also happened and I thought this problem and version issue have to be fixed.

Is there any way to read RIEGELI files easily?

Duy V. Phung · Answer 1 · Thu Mar 10 2022 21:57:57 GMT+0800 (China Standard Time)

I just resolved this problem, you should make sure your gcc is a recent new version (my system with gcc-7.5 on Ubuntu 16.04).

In terms of python2 you also should install like this:

Remove bazel and reinstall

bazel clean --expunge
rm -rf ~/.cache/bazel

You can re-install follow this instruction

Install python2 dependency

sudo apt update && sudo apt install python-dev

I hope this can help you! Good luck!

David Choi · Answer 2 · Tue Mar 22 2022 00:40:41 GMT+0800 (China Standard Time)

Do the above instructions work for you? In particular, you need to install both Python 3 and Python 2, and this is a not a problem that can be fixed in the repo itself.

Duy V. Phung · Answer 3 · Tue Mar 22 2022 07:25:42 GMT+0800 (China Standard Time)

@davidhchoi I get this when I try to run execution/evalutation code, it's seems missing this file contest_problem.pb.h

Can you check this one?

karlie38 · Answer 4 · Tue Mar 22 2022 08:04:07 GMT+0800 (China Standard Time)

My python version is 3.x and my gcc version is latest.
I also tested it in another environment (Ubuntu. 18.04 & Python version 3.x ). It worked when I followed the guide. However, in Window environment, it didn't work.

David Choi · Answer 5 · Tue Mar 22 2022 23:42:14 GMT+0800 (China Standard Time)

@karlie38 what is your Python 2 version? Can you check that Python 2 is installed as well as Python 3?

@PhungVanDuy what is your bazel version, exact command line, and local changes (if any)? That is a proto file that should be automatically compiled into a header file by bazel.

Duy V. Phung · Answer 6 · Wed Mar 23 2022 01:33:49 GMT+0800 (China Standard Time)

@davidhchoi My bazel version is bazel-5.0.0-linux-x86_64, can you check this one https://www.kaggle.com/code/duyphung/betacode-build ?

@davidhchoi I saw that many people facing with compiler problem, Can deepmind team release a docker version for everyone? I think it's will help a lot.

David Choi · Answer 7 · Wed Mar 23 2022 02:00:29 GMT+0800 (China Standard Time)

For your first question, is it possible to try this directly on a Linux machine? Or does this happen outside of a notebook? It might be possible that creating the header for the proto fails for some reason (e.g. the build directory is not writeable). I assume load_data_test.cc also fails.

For the second, it's possible, but we might wait to learn a little bit more what issues people run into. I think specifying bazel/python 2/python 3/clang versions covers all issues currently (plus that we can't currently support non-Linux systems).

Duy V. Phung · Answer 8 · Thu Apr 07 2022 17:51:51 GMT+0800 (China Standard Time)

@davidhchoi I have success to build execution code, I have a question that you have any plan to release code for CPP solution testing? As I known that currently this only support Python Solution right?

David Choi · Answer 9 · Thu Apr 07 2022 18:00:08 GMT+0800 (China Standard Time)

The C++ sandbox is significantly more complicated, and more coupled with internal tools, so while we may release it in the future we currently can't promise anything. I think any kind of sandbox will work though; the main purpose of releasing solution testing is to demonstrate how the output checking works in case it isn't clear from the paper, rather than releasing all the sandboxes we use.

That said, I think we'd welcome pull requests to add a C++ sandbox (and also giving instructions for colab, other OSs, a docker image, etc. as asked in your last question)

Duy V. Phung · Answer 10 · Thu Apr 07 2022 18:16:52 GMT+0800 (China Standard Time)

@davidhchoi Thank you for your information, I just need to ask because if we have a complete sandbox for the test maybe it's can help with some ideas to use RL to guide the decoder with execution score as a part of the reward. Maybe I can make docker image for everyone.

David Choi · Answer 11 · Thu Apr 07 2022 19:23:31 GMT+0800 (China Standard Time)

RL to guide the encoder is an interesting idea, but I think Python evaluation should be enough to get you quite far on this so you can see if it's viable before investing in C++.

Duy V. Phung · Answer 12 · Wed Apr 27 2022 05:17:02 GMT+0800 (China Standard Time)

Hi @davidhchoi ,

I am working on few ideas to improve AlphaCode so I have few research question that I want to discuss with you, if you don't mind can you give me your contact? It would be my pleasure to speak to you directly via email/discord or slack. Looking for your response.

Thanks.

David Choi · Answer 13 · Thu Apr 28 2022 16:58:49 GMT+0800 (China Standard Time)

Sure. My contact information is in the AlphaCode paper. You can reach me at davidhchoi at deepmind.com.

Duy V. Phung · Answer 14 · Sat Apr 30 2022 08:07:47 GMT+0800 (China Standard Time)

@davidhchoi thank you for your information, I have sent you an email via duyphung.ai@gmail.com. Looking for your response.

Danial Zohairen · Answer 15 · Wed May 25 2022 23:05:57 GMT+0800 (China Standard Time)

Do the above instructions work for you? In particular, you need to install both Python 3 and Python 2, and this is a not a problem that can be fixed in the repo itself.

I already have installed both python2 and python3 and I'm still getting the same error. I'm working on a windows10

karlie38 · Answer 16 · Thu May 26 2022 09:46:50 GMT+0800 (China Standard Time)

@DanialZohairen It didn't work out for me in window environment. I did it in Ubuntu. 18.04 & Python version 3.x ).

David Choi · Answer 17 · Thu May 26 2022 19:20:30 GMT+0800 (China Standard Time)

Yes, as mentioned in https://github.com/deepmind/code_contests#supported-platforms we have only tested this on Linux, and aren't able to test this on other platforms like Windows. If someone manages to get it working on Windows though, we'd be happy to update the documentation with instructions.

Muhammad Aman · Answer 18 · Sun May 29 2022 08:23:03 GMT+0800 (China Standard Time)

@karlie38 may I know which versions of gcc and bazel you used? I'm trying to create a docker image for this but I keep getting this error:

karlie38 · Answer 19 · Mon May 30 2022 19:45:55 GMT+0800 (China Standard Time)

My system has updated and i'm not sure about gcc version, but bazel version was bazel-5.0.0 and gcc version was 9.3.0. @LaudemPax

Muhammad Aman · Answer 20 · Thu Jun 02 2022 08:32:45 GMT+0800 (China Standard Time)

In case anyone else needs it, I made a docker container as a workaround to get the bazel build working on Windows. Its on Docker Hub and has been used to train a model with the code_contests dataset on windows.

wvaughn409 · Answer 21 · Tue Jun 07 2022 13:02:08 GMT+0800 (China Standard Time)

In case anyone else needs it, I made a docker container as a workaround to get the bazel build working on Windows. Its on Docker Hub and has been used to train a model with the code_contests dataset on windows.

**YET ANOTHER UPDATE: I decided to just spin up an ubuntu vm and try this the right way, and I'm encountering the same error as with the Docker container, so I have created a new issue:
#17

**NEW UPDATE: next problem I'm getting is that when I attempt to perform the execution/evaluation sample code like so I get an error:

docker run -it -v ${PWD}:/build -w /build laudempax/deepmind-bazel:ubuntu-20.04 run -c opt execution:solve_example /build/tmp/dm-code_contests/code_contests_valid.riegeli

Target //execution:solve_example up-to-date: bazel-bin/execution/solve_example INFO: Elapsed time: 92.776s, Critical Path: 23.83s INFO: 1001 processes: 320 internal, 681 processwrapper-sandbox. INFO: Build completed successfully, 1001 total actions INFO: Build completed successfully, 1001 total actions Failed: Gregor and Cryptography problem not found. Did you pass the validation dataset?

**UPDATE: I was able to get this working by running it this way in elevated Powershell (from the folder with the WORKSPACE file):

docker run -it -v ${PWD}:/build -w /build laudempax/deepmind-bazel:ubuntu-20.04 run -c opt :print_names_and_sources /build/tmp/dm-code_contests/code_contests_valid.riegeli

I tried out your Docker container, thank you by the way for doing that, but the command cannot seem to find the local .riegeli file I'm referencing:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\code_contests-main/tmp/dm-code_contests/code_contests_valid.riegeli'

Here is the modified command I executed from terminal:
docker run -it -v %cd%:/build -w /build laudempax/deepmind-bazel:ubuntu-20.04 run -c opt :print_names_and_sources %cd%/tmp/dm-code_contests/code_contests_valid.riegeli

C:\code_contests-main>docker run -it -v %cd%:/build -w /build laudempax/deepmind-bazel:ubuntu-20.04 run -c opt :print_names_and_sources %cd%/tmp/dm-code_contests/code_contests_valid.riegeli Extracting Bazel installation... Starting local Bazel server and connecting to it... DEBUG: Rule 'com_github_grpc_grpc' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "987ea637191b99f61ef24c54315c6eb7457eac11107513da1562bd2213163bde" DEBUG: Repository com_github_grpc_grpc instantiated at: /build/WORKSPACE:5:13: in <toplevel> Repository rule http_archive defined at: /root/.cache/bazel/_bazel_root/7b7747ec045ae606eb720a1222f56098/external/bazel_tools/tools/build_defs/repo/http.bzl:353:31: in <toplevel> INFO: Analyzed target //:print_names_and_sources (86 packages loaded, 2023 targets configured). INFO: Found 1 target... INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/any_pb2.py: external/com_google_protobuf/python: warning: directory does not exist. INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/descriptor_pb2.py: external/com_google_protobuf/python: warning: directory does not exist. INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/timestamp_pb2.py: external/com_google_protobuf/python: warning: directory does not exist. INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/wrappers_pb2.py: external/com_google_protobuf/python: warning: directory does not exist. INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/struct_pb2.py: external/com_google_protobuf/python: warning: directory does not exist. INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/source_context_pb2.py: external/com_google_protobuf/python: warning: directory does not exist. INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/duration_pb2.py: external/com_google_protobuf/python: warning: directory does not exist. INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/field_mask_pb2.py: external/com_google_protobuf/python: warning: directory does not exist. INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/compiler/plugin_pb2.py: external/com_google_protobuf/python: warning: directory does not exist. INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/type_pb2.py: external/com_google_protobuf/python: warning: directory does not exist. INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/api_pb2.py: external/com_google_protobuf/python: warning: directory does not exist. INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/empty_pb2.py: external/com_google_protobuf/python: warning: directory does not exist. INFO: From ProtoCompile external/com_google_riegeli/python/riegeli/records/records_metadata_pb2.py: external/com_google_protobuf/python: warning: directory does not exist. INFO: From Compiling riegeli/records/record_reader.cc: In file included from external/com_google_absl/absl/container/inlined_vector.h:54, from external/com_google_absl/absl/strings/cord.h:78, from external/com_google_riegeli/riegeli/records/record_reader.h:27, from external/com_google_riegeli/riegeli/records/record_reader.cc:15: external/com_google_absl/absl/container/internal/inlined_vector.h: In constructor 'riegeli::RecordReaderBase::RecordReaderBase()': external/com_google_absl/absl/container/internal/inlined_vector.h:433:5: warning: '<anonymous>.absl::lts_20211102::inlined_vector_internal::Storage<int, 1, std::allocator<int> >::data_' is used uninitialized in this function [-Wuninitialized] 433 | data_ = other_storage.data_; | ^~~~~ external/com_google_absl/absl/container/internal/inlined_vector.h: In constructor 'riegeli::RecordReaderBase::RecordReaderBase(riegeli::Closed)': external/com_google_absl/absl/container/internal/inlined_vector.h:433:5: warning: '<anonymous>.absl::lts_20211102::inlined_vector_internal::Storage<int, 1, std::allocator<int> >::data_' is used uninitialized in this function [-Wuninitialized] 433 | data_ = other_storage.data_; | ^~~~~ external/com_google_absl/absl/container/internal/inlined_vector.h: In member function 'void riegeli::RecordReaderBase::Reset(riegeli::Closed)': external/com_google_absl/absl/container/internal/inlined_vector.h:433:5: warning: '<anonymous>.absl::lts_20211102::inlined_vector_internal::Storage<int, 1, std::allocator<int> >::data_' is used uninitialized in this function [-Wuninitialized] 433 | data_ = other_storage.data_; | ^~~~~ Target //:print_names_and_sources up-to-date: bazel-bin/print_names_and_sources INFO: Elapsed time: 149.233s, Critical Path: 30.23s INFO: 716 processes: 21 internal, 695 processwrapper-sandbox. INFO: Build completed successfully, 716 total actions INFO: Build completed successfully, 716 total actions Traceback (most recent call last): File "/root/.cache/bazel/_bazel_root/7b7747ec045ae606eb720a1222f56098/execroot/__main__/bazel-out/k8-opt/bin/print_names_and_sources.runfiles/__main__/print_names_and_sources.py", line 46, in <module> _print_names_and_sources(sys.argv[1:]) File "/root/.cache/bazel/_bazel_root/7b7747ec045ae606eb720a1222f56098/execroot/__main__/bazel-out/k8-opt/bin/print_names_and_sources.runfiles/__main__/print_names_and_sources.py", line 39, in _print_names_and_sources for problem in _all_problems(filenames): File "/root/.cache/bazel/_bazel_root/7b7747ec045ae606eb720a1222f56098/execroot/__main__/bazel-out/k8-opt/bin/print_names_and_sources.runfiles/__main__/print_names_and_sources.py", line 32, in _all_problems reader = riegeli.RecordReader(io.FileIO(filename, mode='rb'),) FileNotFoundError: [Errno 2] No such file or directory: 'C:\\code_contests-main/tmp/dm-code_contests/code_contests_valid.riegeli'

Yanning Zhou · Answer 22 · Fri Mar 24 2023 17:04:03 GMT+0800 (China Standard Time)

The C++ sandbox is significantly more complicated, and more coupled with internal tools, so while we may release it in the future we currently can't promise anything. I think any kind of sandbox will work though; the main purpose of releasing solution testing is to demonstrate how the output checking works in case it isn't clear from the paper, rather than releasing all the sandboxes we use.

That said, I think we'd welcome pull requests to add a C++ sandbox (and also giving instructions for colab, other OSs, a docker image, etc. as asked in your last question)

Hi!
Is there any plan to release the c++ sandbox currently?