Download_wbt.py Error
CarsonPruitt-NOAA opened this issue · comments
- whitebox version: 2.2.0
- Python version: 3.8.10
- Operating System: Ubuntu 20.04
Description
I'm trying to download the WBT binaries for use in a docker container. I would like to be able to have them set up in the docker image so that each container will not have to download them individually.
What I Did
Here's what I'm running inside my terminal to try to download the binaries and the error it produces:
# python /usr/local/lib/python3.8/dist-packages/whitebox/download_wbt.py
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py", line 388, in get_provider
module = sys.modules[moduleOrReq]
KeyError: 'whitebox'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/whitebox/download_wbt.py", line 182, in <module>
download_wbt()
File "/usr/local/lib/python3.8/dist-packages/whitebox/download_wbt.py", line 20, in download_wbt
pkg_resources.resource_filename(package_name, "whitebox_tools.py")
File "/usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py", line 1201, in resource_filename
return get_provider(package_or_requirement).get_resource_filename(
File "/usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py", line 390, in get_provider
__import__(moduleOrReq)
File "/usr/local/lib/python3.8/dist-packages/whitebox/whitebox.py", line 5, in <module>
from .whitebox_tools import WhiteboxTools
ImportError: attempted relative import with no known parent package
However, opening a python interpreter and running from whitebox import WhiteboxTools; WhiteboxTools() will successfully download the binaries.
You can download the WBT binary from this direct link. Unzip it to the whitebox package root directory. Move the WBT binary whitebox_tools from the WBT folder to the whitebox root directory. The file structure should like this. In that way, the docker container won't need to download the WBT binary every time.
Yes thank you. I got your email response last night from my discussion post. Carson and I work together and try the notes from the discussion post and will keep you posted.
Our Ubuntu 20.04 pathing is /usr/local/lib/python3.8/dist-packages./whitebox and not the path you have above. I assume that is ok. Also... I experimented with the .set_whitebox_dir methods a few times, and set it to both the /usr/local/lib/python3.8/dist-packages./whitebox and also /usr/local/lib/python3.8/dist-packages./whitebox/WBT But no luck. Did I path that correctly?
Other than the whitebox_tools requires GLIBC 2.32 or higher I mentioned in discussion 55 which is very illogical to me. I wonder if is something going wrong we can not see. I am going to debug the actual whitebox_tools.py file in the dist-packages today and see if that tells us something.
We also carefully watched permissions on the folders and that didn't work.
The most bizarre part of this story is that all of our servers started failing at the same time. We don't know when exactly it started failing but likely between Thurs and Monday morning. We are able to replicate fail on multiple servers, inside and outside out network, delaminating environmental network changes and possibly even delimiting some system upgrades (but not 100% confirmed on upgrades not being a culprit). At this point, our software is non-functional and we are trying to figure out why it happened and how to get our systems running again.
See jblindsay/whitebox-tools#338 (comment).
The issue is because the latest WBT binary for Linux is compiled on Ubuntu 22.x, which requires a newer GLIBC. We need it be compiled on Ubuntu 20.x so that it can be compatible with older Ubuntu OS.
You can download an older version of WBT binary I complied on Ubuntu 18.04 from here. It should work with Ubuntu 20.04. Follow the steps I mentioned above the replace the WBT binary in your whitebox package folder. Otherwise, it always pull the latest binary from whiteboxgeo.com, which is incimpatible with Ubuntu 20.04.
I am trying to compile WBT from source on Ubuntu 20.04. If successful, I can then send the binary to @jblindsay to replace it on the whiteboxgeo.com server. That could resolve all the GLIBC issues.
ah.. so they must have changed it in the past week or so then (I am guessing). Yes, please compile and have it put on the whiteboxgeo.com server. We will try it on our side as well.
You are correct. The WhiteboxTools Open Core v2.3.0 was released yesterday, and that's why the files on the whiteboxgeo.com have been replaced.
ok. Can you keep us posted when you have an Ubuntu 20.04 version posted? We will hang tight.
I tried to build WBT from sources on a fresh Ubuntu 20.04 and encountered various errors. I was able to some of the errors but I am now stuck on this one.
@williamlidberg Did you build WBT from sources on Ubuntu 20.04 successfully? Any advice on this issue?
This is precisely why I have also distributed the musl-compiled version of Wbt. Please see here. Under the title, 'Is this the wrong version? Download the correct version below.' you should see 'Linux version (compiled with musl)'.
@jblindsay Awesome! I did not that that it existed. I just tested it with Google Colab, and it works. Will the musl-compiled version work on all Linux distributions? Should pip install whitebox users download the musl-compiled version by default?
I have released v2.3.1 for resolving this issue.
If you encounter any GLIBC errors after installing the whitebox package, you can try the following command:
import whitebox
whitebox.download_wbt(linux_musl=True, reset=True)Alternatively, you can set the environment variable WBT_LINUX to MUSL before installing the whitebox package. It will automatically download the MUSL version of WhiteboxTools. This is a better solution if you are using docker.
import os
os.environ["WBT_LINUX"] = "MUSL"