Cant get it to run! Numba version...
Gitterman69 opened this issue · comments
If anyone can get this to work (or create a whl)???
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts numba==0.48.0
Good luck. It needs this older version of numba to work, but the older version causes all sorts of hassles trying to cooperate with the other packages.
Any way to fix this???
The problem is the requrements misses a lot of required packages to run the script.
This is as close as I can get which is many more packages than theirs.
python -m pip install --upgrade pip
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts wheel==0.38.4
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts git+https://github.com/jmoso13/jukebox-diffusion@baeee35ee0198e8d5954d07519bf0ea6a9a66719
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts fire==0.1.3
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts soundfile==0.10.3.post1
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts unidecode==1.1.1
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts numba==0.48.0
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts librosa==0.7.2
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts mpi4py==3.1.4
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts av==10.0.0
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts git+https://github.com/jmoso13/a-unet.git@add_dilation
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts git+https://github.com/jmoso13/audio-diffusion-pytorch.git@noise_pass
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts torchmetrics==0.11.4
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts pytorch-lightning==1.7
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts wandb==0.15.11
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts Pygments==2.14.0
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts git+https://github.com/arogozhnikov/einops.git@4f6afb3eb368661c383b2a7717d416814c1afc7f
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts ipython==8.15.0
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts pandas==2.1.1
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts resampy==0.3.1
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts tqdm==4.57.0
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts pillow==10.0.1
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts PyYAML==6.0.1
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts git+https://github.com/Harmonai-org/sample-generator.git@c61cc65257ae8c178620ddc37c04aa24b421b326
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts git+https://github.com/crowsonkb/v-diffusion-pytorch.git@439ed4caf765ec6f8fccb52213a859cac9e76d3b
pip uninstall -y torch
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts torch==2.0.1+cu118 torchvision==0.15.2+cu118 torchaudio==2.0.2+cu118 --index-url https://download.pytorch.org/whl/cu118
The problem is that old version of numba will not compile/build with the rest that are more recent and that VoC uses.
If numba could be updated to the newest version it would be OK, but that needs tweaking the rest of them and probbaly the scripts code.
Hi, The above was a repost from one of my users from my Discord as I am tyring to get Jukebox Diffusion working in Visions of Chaos (https://softology.pro/voc.htm)
I started from your requirements which did not specify versions required and did not seem to include all required packages. So as I went along testing and adding other needed packages, I got to the requirments as shown above.
The only problem is that your scripts seem to require an older version of numba 48.0 way back from Jan 2020. It also needs an older numpy version. So I try numpy 1.22.4 and then numba 48.0. numba won't compile with
INFO: No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils
building 'numba._dynfunc' extension
creating build\temp.win-amd64-cpython-310
creating build\temp.win-amd64-cpython-310\Release
creating build\temp.win-amd64-cpython-310\Release\numba
INFO: C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\bin\HostX86\x64\cl.exe /c /nologo /O2 /W3 /GL /DNDEBUG /MD -ID:\Tests\Jukebox Diffusion\voc_jukeboxdiffusion\include -ID:\Python\include -ID:\Python\Include -IC:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\include -IC:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\ATLMFC\include -IC:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\VS\include -IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um /Tcnumba/_dynfuncmod.c /Fobuild\temp.win-amd64-cpython-310\Release\numba/_dynfuncmod.obj -g
cl : Command line warning D9002 : ignoring unknown option '-g'
Maybe it also need an older VS version too?
Updating numpy and numba gets all the requirments installed. So now I have the following in a new environment.
python -m pip install --upgrade pip
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts wheel==0.38.4
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts git+https://github.com/jmoso13/jukebox-diffusion@baeee35ee0198e8d5954d07519bf0ea6a9a66719
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts fire==0.1.3
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts soundfile==0.10.3.post1
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts unidecode==1.1.1
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts numpy==1.26.0
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts numba==0.58.0
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts librosa==0.7.2
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts mpi4py==3.1.4
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts av==10.0.0
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts git+https://github.com/jmoso13/a-unet.git@add_dilation
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts git+https://github.com/jmoso13/audio-diffusion-pytorch.git@noise_pass
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts torchmetrics==0.11.4
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts pytorch-lightning==1.7
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts wandb==0.15.11
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts Pygments==2.14.0
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts git+https://github.com/arogozhnikov/einops.git@4f6afb3eb368661c383b2a7717d416814c1afc7f
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts ipython==8.15.0
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts pandas==2.1.1
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts resampy==0.3.1
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts tqdm==4.57.0
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts pillow==10.0.1
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts PyYAML==6.0.1
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts git+https://github.com/Harmonai-org/sample-generator.git@c61cc65257ae8c178620ddc37c04aa24b421b326
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts git+https://github.com/crowsonkb/v-diffusion-pytorch.git@439ed4caf765ec6f8fccb52213a859cac9e76d3b
pip uninstall -y torch
pip install --no-cache-dir --ignore-installed --force-reinstall --no-warn-conflicts torch==2.0.1+cu118 torchvision==0.15.2+cu118 torchaudio==2.0.2+cu118 --index-url https://download.pytorch.org/whl/cu118
But then when trying to train with the first example command
python train.py --train-data ./wavs --jb-level 2 --ckpt-save-location ./ckpts
I get this error
from numba.decorators import jit as optional_jit
ModuleNotFoundError: No module named 'numba.decorators'
So numba needs to be older for this to work. But older numba is not compatible with newer requirements.
There is no whl for numba 48.0 I can find. If there was I could possibly get Jukebox Diffusion working. Hence my first request if anyone could compile it. Some of my users are good at compiling stubborn WHL files and if I get a working WHL it can be used for any install to work much smoother.
If I then try and uninstall numba and install numba 0.48.0 I get errors like
No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils
RuntimeError: No compatible cmake generator installed on this machine
I do have cmake latest installed which is used to compile other ML/AI systems in Visions of Chaos.
Maybe the current code could be updated to work with the latest numba and numpy?
Uninstalling numba, numpy and librosa completely and then installing the latest versions of all 3, then moving the jukebox\jukebox folder under the root \jukebox gets to this final error.
from mpi4py import MPI # This must be imported in order to get e rrors from all ranks to show up
ImportError: DLL load failed while importing MPI: The specified module could not be found.
Which looking around, MPI seems a pain to compile too. Gotta love Python sometimes. If you do ever get an updated version of the code that works with the latest packages please let me know. Otherwise, feel free to close this mess :)
Thanks so much for reporting this issue, unfortunately this repo relies on an extremely old version of OpenAI's Jukebox along with more modern repos for doing diffusion. The whole thing leads to a mess of requirements and I totally understand the frustration with conflicting package versions. I'm working on a more updated version that does not rely on Jukebox at all, uses updated package versions, and should generate at higher quality. I'll tag you and let you know when that is released so you can try it out! I'm sorry you couldn't get it working and I appreciate you trying. Closing for now.