Installation is a mess, instructions are a mess
magnacartatron opened this issue · comments
ChatGPT helped remove all the swear words.
I've encountered difficulties trying to set this up on Ubuntu, MacOS, and Windows. I've noticed some inconsistencies in the instructions, and it seems some tools and libraries might be either too new or outdated. It would be greatly appreciated if these issues could be addressed to make the project more user-friendly for everyone. Thank you.
We've tested the following installation instructions and pip package. What error message did you encounter, can you post it here?
conda create -y --name openchat
conda activate openchat
conda install -y python
conda install -y cudatoolkit-dev -c conda-forge
pip3 install torch torchvision torchaudio
pip3 install packaging ninja
pip3 install --no-build-isolation "flash-attn<2"
pip3 install ochat
pip 23.3.1 from C:\Python311\Lib\site-packages\pip (python 3.11)
With your commands : First 6 ok,
then :
PS C:\Users\xxxxxxxx\projects>pip3 install --no-build-isolation "flash-attn<2"
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting flash-attn<2
Downloading flash_attn-1.0.9.tar.gz (1.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 7.2 MB/s eta 0:00:00
Preparing metadata (pyproject.toml) ... error
error: subprocess-exited-with-error
× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [34 lines of output]
No CUDA runtime is found, using CUDA_HOME='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.3'
fatal: not a git repository (or any of the parent directories): .git
Warning: Torch did not find available GPUs on this system.
If your intention is to cross-compile, this is not an error.
By default, Apex will cross-compile for Pascal (compute capabilities 6.0, 6.1, 6.2),
Volta (compute capability 7.0), Turing (compute capability 7.5),
and, if the CUDA version is >= 11.0, Ampere (compute capability 8.0).
If you wish to cross-compile for a single specific architecture,
export TORCH_CUDA_ARCH_LIST="compute capability" before running setup.py.
torch.__version__ = 2.1.0+cpu
running dist_info
creating C:\Users\xxxxxxxx\AppData\Local\Temp\pip-modern-metadata-2xvr8d9z\flash_attn.egg-info
writing C:\Users\xxxxxxxx\AppData\Local\Temp\pip-modern-metadata-2xvr8d9z\flash_attn.egg-info\PKG-INFO
writing dependency_links to C:\Users\xxxxxxxx\AppData\Local\Temp\pip-modern-metadata-2xvr8d9z\flash_attn.egg-info\dependency_links.txt
writing requirements to C:\Users\xxxxxxxx\AppData\Local\Temp\pip-modern-metadata-2xvr8d9z\flash_attn.egg-info\requires.txt
writing top-level names to C:\Users\xxxxxxxx\AppData\Local\Temp\pip-modern-metadata-2xvr8d9z\flash_attn.egg-info\top_level.txt
writing manifest file 'C:\Users\xxxxxxxx\AppData\Local\Temp\pip-modern-metadata-2xvr8d9z\flash_attn.egg-info\SOURCES.txt'
reading manifest file 'C:\Users\xxxxxxxx\AppData\Local\Temp\pip-modern-metadata-2xvr8d9z\flash_attn.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.cu' under directory 'flash_attn'
warning: no files found matching '*.h' under directory 'flash_attn'
warning: no files found matching '*.cuh' under directory 'flash_attn'
warning: no files found matching '*.cpp' under directory 'flash_attn'
adding license file 'LICENSE'
adding license file 'AUTHORS'
writing manifest file 'C:\Users\xxxxxxxx\AppData\Local\Temp\pip-modern-metadata-2xvr8d9z\flash_attn.egg-info\SOURCES.txt'
creating 'C:\Users\xxxxxxxx\AppData\Local\Temp\pip-modern-metadata-2xvr8d9z\flash_attn-1.0.9.dist-info'
error: invalid command 'bdist_wheel'
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
and conda -v :
PS C:\Users\xxxxxxxx\projects> conda -v
# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
Traceback (most recent call last):
File "C:\Users\xxxxxxxx\miniconda3\Lib\site-packages\conda\exception_handler.py", line 17, in __call__
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\xxxxxxxx\miniconda3\Lib\site-packages\conda\cli\main.py", line 78, in main_subshell
exit_code = do_call(args, parser)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\xxxxxxxx\miniconda3\Lib\site-packages\conda\cli\conda_argparse.py", line 160, in do_call
relative_mod, func_name = args.func.rsplit(".", 1)
^^^^^^^^^
AttributeError: 'Namespace' object has no attribute 'func'
`$ C:\Users\xxxxxxxx\miniconda3\Scripts\conda-script.py -v`
environment variables:
CIO_TEST=<not set>
CONDA_EXE=C:\Users\xxxxxxxx\miniconda3\condabin\..\Scripts\conda.exe
CONDA_EXES="C:\Users\xxxxxxxx\miniconda3\condabin\..\Scripts\conda.exe"
CONDA_ROOT=C:\Users\xxxxxxxx\miniconda3
CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.3
CURL_CA_BUNDLE=<not set>
HOMEPATH=\Users\xxxxxxxx
LD_PRELOAD=<not set>
PATH=C:\Program Files\NVIDIA GPU Computing
Toolkit\CUDA\v12.3\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\C
UDA\v12.3\libnvvp;C:\Python311\Scripts\;C:\Python311\;C:\Python312\Scr
ipts\;C:\Python312\;C:\Windows\system32;C:\Windows;C:\Windows\System32
\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\
OpenSSH\;C:\ProgramData\chocolatey\bin;C:\Program
Files\nodejs\;C:\Program Files\Git\cmd;C:\Program Files\NVIDIA
Corporation\Nsight Compute 2023.3.0\;C:\Program Files (x86)\NVIDIA
Corporation\PhysX\Common;C:\Users\xxxxx
xxxxx\miniconda3;C:\Users\xxxxx
xxxxx\miniconda3\Library\mingw-w64\bin;C:\Users\xxxxx
xxxxx\miniconda3\Library\usr\bin;C:\Users\xxxxx
xxxxx\miniconda3\Library\bin;C:\Users\xxxxx
xxxxx\miniconda3\Scripts;C:\Users\xxxxx
xxxxx\.cargo\bin;C:\Users\xxxxx
xxxxx\AppData\Local\Microsoft\WindowsApps;C:\Users\xxxxx
xxxxx\AppData\Roaming\npm
PSMODULEPATH=C:\Users\xxxxx
xxxxx\Documents\WindowsPowerShell\Modules;C:\Program Files\WindowsPow
erShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules
REQUESTS_CA_BUNDLE=<not set>
SSL_CERT_FILE=<not set>
active environment : None
user config file : C:\Users\xxxxxxxx\.condarc
populated config files :
conda version : 23.9.0
conda-build version : not installed
python version : 3.11.5.final.0
virtual packages : __archspec=1=x86_64
__cuda=12.3=0
__win=0=0
base environment : C:\Users\xxxxxxxx\miniconda3 (writable)
conda av data dir : C:\Users\xxxxxxxx\miniconda3\etc\conda
conda av metadata url : None
channel URLs : https://repo.anaconda.com/pkgs/main/win-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/win-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/msys2/win-64
https://repo.anaconda.com/pkgs/msys2/noarch
package cache : C:\Users\xxxxxxxx\miniconda3\pkgs
C:\Users\xxxxxxxx\.conda\pkgs
C:\Users\xxxxxxxx\AppData\Local\conda\conda\pkgs
envs directories : C:\Users\xxxxxxxx\miniconda3\envs
C:\Users\xxxxxxxx\.conda\envs
C:\Users\xxxxxxxx\AppData\Local\conda\conda\envs
platform : win-64
user-agent : conda/23.9.0 requests/2.31.0 CPython/3.11.5 Windows/10 Windows/10.0.22621
administrator : True
netrc file : None
offline mode : False
An unexpected error has occurred. Conda has prepared the above report.
If you suspect this error is being caused by a malfunctioning plugin,
consider using the --no-plugins option to turn off plugins.
Example: conda --no-plugins install <package>
Alternatively, you can set the CONDA_NO_PLUGINS environment variable on
the command line to run the command without plugins enabled.
Example: CONDA_NO_PLUGINS=true conda install <package>
If submitted, this report will be used by core maintainers to improve
future releases of conda.
Would you like conda to send this report to the core maintainers? [y/N]: N
No report sent. To permanently opt-out, use
$ conda config --set report_errors false
help pls
I successfully install it on WSL 2 (Windows 11) with following step:
Environment
- I cannot make it works on my windows 11 (failed on installing (build)
vLLM), but - Windows 11, WSL 2, Anaconda works fine
Here is the Environment in WSL
(openchat) doodlebear@BEAR-PC:/mnt/c/Users/admin$ conda -v
# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
Traceback (most recent call last):
File "/home/doodlebear/miniconda3/lib/python3.11/site-packages/conda/exception_handler.py", line 17, in __call__
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/doodlebear/miniconda3/lib/python3.11/site-packages/conda/cli/main.py", line 78, in main_subshell
exit_code = do_call(args, parser)
^^^^^^^^^^^^^^^^^^^^^
File "/home/doodlebear/miniconda3/lib/python3.11/site-packages/conda/cli/conda_argparse.py", line 160, in do_call
relative_mod, func_name = args.func.rsplit(".", 1)
^^^^^^^^^
AttributeError: 'Namespace' object has no attribute 'func'
`$ /home/doodlebear/miniconda3/bin/conda -v`
environment variables:
CIO_TEST=<not set>
CONDA_DEFAULT_ENV=openchat
CONDA_EXE=/home/doodlebear/miniconda3/bin/conda
CONDA_PREFIX=/home/doodlebear/miniconda3/envs/openchat
CONDA_PREFIX_1=/home/doodlebear/miniconda3
CONDA_PROMPT_MODIFIER=(openchat)
CONDA_PYTHON_EXE=/home/doodlebear/miniconda3/bin/python
CONDA_ROOT=/home/doodlebear/miniconda3
CONDA_SHLVL=2
CURL_CA_BUNDLE=<not set>
LD_PRELOAD=<not set>
PATH=/home/doodlebear/miniconda3/envs/openchat/bin:/home/doodlebear/minicon
da3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/
bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Users/admin/Ap
pData/Local/fnm_multishells/2820_1699353565754:/mnt/c/ProgramData/anac
onda3:/mnt/c/ProgramData/anaconda3/Library/mingw-
w64/bin:/mnt/c/ProgramData/anaconda3/Library/usr/bin:/mnt/c/ProgramDat
a/anaconda3/Library/bin:/mnt/c/ProgramData/anaconda3/Scripts:/mnt/c/Pr
ogramData/anaconda3/bin:/mnt/c/ProgramData/anaconda3/condabin:/mnt/c/P
rogram Files/PowerShell/7:/mnt/c/Program Files/NVIDIA GPU Computing
Toolkit/CUDA/v11.8/bin:/mnt/c/Program Files/NVIDIA GPU Computing Toolk
it/CUDA/v11.8/libnvvp:/mnt/c/WINDOWS/system32:/mnt/c/WINDOWS:/mnt/c/WI
NDOWS/System32/Wbem:/mnt/c/WINDOWS/System32/WindowsPowerShell/v1.0:/mn
t/c/WINDOWS/System32/OpenSSH:/mnt/c/Program Files (x86)/NVIDIA
Corporation/PhysX/Common:/mnt/c/Program Files/NVIDIA
Corporation/NVIDIA NvDLISR:/mnt/c/Program
Files/Bandizip:/mnt/c/Program
Files/Git/cmd:/mnt/d/code/Environment/cudnn/cudnn-windows-x86_64-
8.9.0.131_cuda12-
archive/bin:/mnt/c/ProgramData/anaconda3/Scripts:/mnt/d/code/Environme
nt/wget:/mnt/c/Program
Files/Docker/Docker/resources/bin:/mnt/c/Program
Files/PowerShell/7:/mnt/c/Program Files (x86)/Windows Kits/10/Windows
Performance Toolkit:/mnt/c/Program
Files/NVIDIA/CUDNN/v8.9.6/bin:/mnt/c/Program Files/NVIDIA
Corporation/Nsight Compute 2022.3.0:/mnt/c/Users/admin/AppData/Local/M
icrosoft/WindowsApps:/mnt/c/Users/admin/AppData/Local/Programs/Microso
ft VS Code/bin:/mnt/c/Users/admin/AppData/Local/Microsoft/WinGet/Packa
ges/Schniz.fnm_Microsoft.Winget.Source_8wekyb3d8bbwe:/mnt/c/Users/admi
n/AppData/Local/Programs/oh-my-
posh/bin:/mnt/c/texlive/2023/bin/windows:/snap/bin
REQUESTS_CA_BUNDLE=<not set>
SSL_CERT_FILE=<not set>
active environment : openchat
active env location : /home/doodlebear/miniconda3/envs/openchat
shell level : 2
user config file : /home/doodlebear/.condarc
populated config files :
conda version : 23.9.0
conda-build version : not installed
python version : 3.11.5.final.0
virtual packages : __archspec=1=x86_64
__cuda=12.3=0
__glibc=2.35=0
__linux=5.15.90.1=0
__unix=0=0
base environment : /home/doodlebear/miniconda3 (writable)
conda av data dir : /home/doodlebear/miniconda3/etc/conda
conda av metadata url : None
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /home/doodlebear/miniconda3/pkgs
/home/doodlebear/.conda/pkgs
envs directories : /home/doodlebear/miniconda3/envs
/home/doodlebear/.conda/envs
platform : linux-64
user-agent : conda/23.9.0 requests/2.31.0 CPython/3.11.5 Linux/5.15.90.1-microsoft-standard-WSL2 ubuntu/22.04.2 glibc/2.35
UID:GID : 1000:1000
netrc file : None
offline mode : False
Install OpenChat on Windows 11 (WSL)
Using the instructions a bit different from above
Same for create conda env
conda create -y --name openchat
conda activate openchat
⭐ Alomst same, specify the python version
// conda install -y python
conda install -y python=3.11
Same for install cudatoolkit in conda
- but want to mention that when I am running this instruction, it takes a while and only show the
\and/while processing and ended with nothing in my PowerShell (terminal), so I have to close the terminal and reopen it.
conda install -y cudatoolkit-dev -c conda-forge
⭐ Alomst same, but specify the torch version.
- I noticed that using conda to install torch will some how install cpu version instead of cuda using
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia - I saw @ugref using
torch.__version__ = 2.1.0+cpu, and I did not facing the same issue when using 2.0.1+cu118 installing flash-attn when using Windows 11 (not in WSL)
// pip3 install torch torchvision torchaudio
pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118
- torch==2.1.0 may works well, did not try, check out other pytorch version here
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
Same
pip3 install packaging ninja
⭐ Before install flash-attn, I need to Install g++ Compiler first. (Basically, GPT-4 teaches me everything)
sudo apt update
sudo apt install build-essential
Same
pip3 install --no-build-isolation "flash-attn<2"
pip3 install ochat
I try almost 6 hours on making it works on Windows but fail, So I try WSL 2
I already using Docker, so there is nothing too much when install openchat on WSL
Simply
- Install Conda (maybe come across some env issue, but GPT-4 got it)
- Little issue when install some python package, just ask GPT-4
- Stuck on running the server for a while, if you are not using cluster with multiple GPUs, and like me using a Desktop RTX 3090.
Instead of using
python -m ochat.serving.openai_api_server --model openchat/openchat_3.5 --engine-use-ray --worker-use-ray
Use this just works for me
python -m ochat.serving.openai_api_server --model openchat/openchat_3.5 --tensor-parallel-size 1
Use this to see more parameters settings
python -m ochat.serving.openai_api_server --model openchat/openchat_3.5 -h
Test with request
@imoneoi
I am wondering if a Run Server section right after Installation section will be better?
For example
Run Server
For single GPT (e.g. RTX 3090, 4090)
python -m ochat.serving.openai_api_server --model openchat/openchat_3.5 --tensor-parallel-size 1
For cluster with multiple GPUs
python -m ochat.serving.openai_api_server --model openchat/openchat_3.5 --engine-use-ray --worker-use-ray
For more parameters settings
python -m ochat.serving.openai_api_server --model openchat/openchat_3.5 -h
Original README has run server instruction above, instead of right after the installation
Just made a PR #70