Rebooting kaggle notebook instance with minimal number of packages

Question

Rebooting kaggle notebook instance with minimal number of packages

realharryhero opened this issue a month ago · comments

Hello! For a competition, I want to use tools that are far outside those that are normally used, say for the most recent math-like competition, including LEAN and many scripts. But for example, trying to use LEAN on the kaggle notebook with this code causes some errors:

Code

cd /kaggle/working

sudo apt-get update

set -exo pipefail

sudo apt-get install -y --install-suggests git libgmp-dev cmake ccache clang

bash -c 'curl https://raw.githubusercontent.com/leanprover/elan/master/elan-init.sh -sSf | sh -s -- -y --default-toolchain leanprover/lean4:v4.7.0 || (echo && read -n 1 -s -r -p "Install failed, press ENTER to continue...")' 

source ~/.profile

elan self update

lake_do () {
   lake update
   lake build
}

lake new my_project
cd my_project
lake update
lake build
echo 'got here'
printf '\nrequire mathlib from /kaggle/working/mathlib4' >> ./lakefile.lean
printf 'leanprover/lean4:v4.7.0' > ./lean-toolchain
cat ./lakefile.lean
lake update
lake build
printf '\n\nimport Mathlib.Topology.Basic\n\n#check TopologicalSpace' >> ./Main.lean
lake update
lake build
lake env lean --run ./Main.lean

Various other (seemingly unrelated) issues pop up with other tools I'd like to use, and I believe this is due to the extremely large amount of packages uploaded in each kaggle notebook that would normally be used for commonly-used ML packages like torch or tensorflow, etc. Is there a way to reboot a kaggle notebook into a sort-of "minimal instance" without so many packages? Thank you!

stevemessick · Answer 1 · Wed Jun 26 2024 01:05:27 GMT+0800 (China Standard Time)

@realharryhero I don't understand your code. What language is it written in? What errors are you seeing?

Harry Fan · Answer 2 · Sat Jul 06 2024 20:38:14 GMT+0800 (China Standard Time)

The hidden code is written in the bash language, meant to be run as a shell script. The script must be run on the GPU, but otherwise it can be directly run without any previous notebook cells having run before.
The programming language LEAN is installed through a bash command: bash -c 'curl https://raw.githubusercontent.com/leanprover/elan/master/elan-init.sh -sSf | sh -s -- -y --default-toolchain leanprover/lean4:v4.7.0 || (echo && read -n 1 -s -r -p "Install failed, press ENTER to continue...")', that adds the commands lean, elan and lake to the path of runnable executables.
lake is a portmanteau of lean and make, and lake acts as a command to "build" the LEAN code, much as how one would "build" C code to run it.
Just as there can be issues building C code, there can also be issues building LEAN code - in this case, the first lake build command raises an error, shown below:

Log

✖ [4/7] Building MyProject.Basic:c.o
trace: .> /root/.elan/toolchains/leanprover--lean4---stable/bin/leanc -c -o ././.lake/build/ir/MyProject/Basic.c.o.export ././.lake/build/ir/MyProject/Basic.c -O3 -DNDEBUG -DLEAN_EXPORTING
info: stderr:
/root/.elan/toolchains/leanprover--lean4---stable/bin/clang: symbol lookup error: /root/.elan/toolchains/leanprover--lean4---stable/bin/../lib/libclang-cpp.so.15: undefined symbol: _ZTVN4llvm2cl11OptionValueINSt3__112basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEEEE, version LLVM_15
error: external command '/root/.elan/toolchains/leanprover--lean4---stable/bin/leanc' exited with code 127
✖ [5/7] Building MyProject:c.o
trace: .> /root/.elan/toolchains/leanprover--lean4---stable/bin/leanc -c -o ././.lake/build/ir/MyProject.c.o.export ././.lake/build/ir/MyProject.c -O3 -DNDEBUG -DLEAN_EXPORTING
info: stderr:
/root/.elan/toolchains/leanprover--lean4---stable/bin/clang: symbol lookup error: /root/.elan/toolchains/leanprover--lean4---stable/bin/../lib/libclang-cpp.so.15: undefined symbol: _ZTVN4llvm2cl11OptionValueINSt3__112basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEEEE, version LLVM_15
error: external command '/root/.elan/toolchains/leanprover--lean4---stable/bin/leanc' exited with code 127
✖ [6/7] Building Main:c.o
trace: .> /root/.elan/toolchains/leanprover--lean4---stable/bin/leanc -c -o ././.lake/build/ir/Main.c.o.export ././.lake/build/ir/Main.c -O3 -DNDEBUG -DLEAN_EXPORTING
info: stderr:
/root/.elan/toolchains/leanprover--lean4---stable/bin/clang: symbol lookup error: /root/.elan/toolchains/leanprover--lean4---stable/bin/../lib/libclang-cpp.so.15: undefined symbol: _ZTVN4llvm2cl11OptionValueINSt3__112basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEEEE, version LLVM_15
error: external command '/root/.elan/toolchains/leanprover--lean4---stable/bin/leanc' exited with code 127
Some builds logged failures:
- MyProject.Basic:c.o
- MyProject:c.o
- Main:c.o
error: build failed

The full message after running the bash script is below:

Log

Get:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease [1581 B]
Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease                         
Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB]        
Get:4 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Packages [1672 kB]
Get:5 https://packages.cloud.google.com/apt gcsfuse-focal InRelease [1225 B]   
Get:6 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB]      
Get:7 https://packages.cloud.google.com/apt cloud-sdk InRelease [1616 B]       
Hit:8 http://archive.ubuntu.com/ubuntu focal-backports InRelease               
Get:9 https://packages.cloud.google.com/apt google-fast-socket InRelease [1071 B]
Get:10 https://packages.cloud.google.com/apt gcsfuse-focal/main amd64 Packages [24.1 kB]
Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [4228 kB]
Get:12 https://packages.cloud.google.com/apt cloud-sdk/main amd64 Packages [3089 kB]
Get:13 https://packages.cloud.google.com/apt cloud-sdk/main all Packages [1475 kB]
Get:14 https://packages.cloud.google.com/apt google-fast-socket/main amd64 Packages [15.4 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [33.5 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3882 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1530 kB]
Get:18 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1245 kB]
Get:19 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3758 kB]
Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [30.9 kB]
Get:21 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3733 kB]
Fetched 25.0 MB in 3s (9648 kB/s)                          
Reading package lists... Done
+ bash -c 'curl https://raw.githubusercontent.com/leanprover/elan/master/elan-init.sh -sSf | sh -s -- -y --default-toolchain leanprover/lean4:stable || (echo && read -n 1 -s -r -p "Install failed, press ENTER to continue...")'
info: downloading installer
info: syncing channel updates for 'stable'
info: latest update on stable, lean version v4.9.0
info: downloading component 'lean'
info: installing component 'lean'
info: default toolchain set to 'leanprover/lean4:stable'

  leanprover/lean4:stable installed - Lean (version 4.9.0, x86_64-unknown-linux-gnu, commit 8f9843a4a5fe, Release)

+ source /root/.profile
++ '[' /usr/bin/bash ']'
++ '[' -f /root/.bashrc ']'
++ . /root/.bashrc
+++ '[' -z '' ']'
+++ return
++ mesg n
++ true
++ export PATH=/root/.elan/bin:/opt/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ PATH=/root/.elan/bin:/opt/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ lake new my_project
+ cd my_project
+ lake update
info: my_project: no previous manifest, creating one from scratch
+ lake build
✖ [3/7] Building MyProject.Basic:c.o
trace: .> /root/.elan/toolchains/leanprover--lean4---stable/bin/leanc -c -o ././.lake/build/ir/MyProject/Basic.c.o.export ././.lake/build/ir/MyProject/Basic.c -O3 -DNDEBUG -DLEAN_EXPORTING
info: stderr:
/root/.elan/toolchains/leanprover--lean4---stable/bin/clang: symbol lookup error: /root/.elan/toolchains/leanprover--lean4---stable/bin/../lib/libclang-cpp.so.15: undefined symbol: _ZTVN4llvm2cl11OptionValueINSt3__112basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEEEE, version LLVM_15
error: external command '/root/.elan/toolchains/leanprover--lean4---stable/bin/leanc' exited with code 127
✖ [4/7] Building MyProject:c.o
trace: .> /root/.elan/toolchains/leanprover--lean4---stable/bin/leanc -c -o ././.lake/build/ir/MyProject.c.o.export ././.lake/build/ir/MyProject.c -O3 -DNDEBUG -DLEAN_EXPORTING
info: stderr:
/root/.elan/toolchains/leanprover--lean4---stable/bin/clang: symbol lookup error: /root/.elan/toolchains/leanprover--lean4---stable/bin/../lib/libclang-cpp.so.15: undefined symbol: _ZTVN4llvm2cl11OptionValueINSt3__112basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEEEE, version LLVM_15
error: external command '/root/.elan/toolchains/leanprover--lean4---stable/bin/leanc' exited with code 127
✖ [6/7] Building Main:c.o
trace: .> /root/.elan/toolchains/leanprover--lean4---stable/bin/leanc -c -o ././.lake/build/ir/Main.c.o.export ././.lake/build/ir/Main.c -O3 -DNDEBUG -DLEAN_EXPORTING
info: stderr:
/root/.elan/toolchains/leanprover--lean4---stable/bin/clang: symbol lookup error: /root/.elan/toolchains/leanprover--lean4---stable/bin/../lib/libclang-cpp.so.15: undefined symbol: _ZTVN4llvm2cl11OptionValueINSt3__112basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEEEE, version LLVM_15
error: external command '/root/.elan/toolchains/leanprover--lean4---stable/bin/leanc' exited with code 127
Some builds logged failures:
- MyProject.Basic:c.o
- MyProject:c.o
- Main:c.o
error: build failed

The error message obtained, moreover, is quite cryptic, and doesn't quite show how to solve it (I looked on StackOverflow for quite a while; the solutions in each case an error remotely similar to this occurred, I believe, were platform-specific (if any appeared at all, maybe that was me trying to use another not-so-well-known tool in a kaggle notebook). Moreover, if this error is due to one of the many, many pre-installed packages on the kaggle notebooks, then I may have to search through every package that's causing issues - that would take far too much time, and based on how relatively unknown the tools are, this could theoretically occur for every tool. Something similar occurred when trying to use the InternLM-Math model. Hence, it'd be better to have a sort-of "minimal instance" instead of trying to debug the error message for a few more hours.

Of course, this raises the question of why to go through so much trouble in the first place. My answer is that the AI Mathematical Olympiad Progress Prize 1 has just recently ended, and many new progress prizes will come out (as the grand prize still hasn't been reached yet), and one of the most important things here is to set up a theorem-proving language which can "formalize" mathematics, so the AI can get feedback to improve much like a coding AI can get feedback from running the code to improve its accuracy from ~50% to ~90%.

Moreover, making a sort of "minimal instance" would not just be faster for me, but also perhaps for others competing in the contest, so that they don't have to deal with the slowdown of days spent setting up packages; it could be the difference being or not being in the top 5. Thank you!

stevemessick · Answer 3 · Tue Jul 09 2024 01:10:59 GMT+0800 (China Standard Time)

@realharryhero Thanks for the additional info! What you're asking for is a fairly large change to the platform. It isn't something that can be addressed by kaggle-api. To get better visibility of your request, I suggest you post a topic in our product feedback forum:
https://www.kaggle.com/discussions/product-feedback

Doing so will also make it more visible to other Kaggle users, which might help with allocation of engineering resources.