🚀 Supercharge your LM Studio with a Vector Database!
‼️ 🐍Python 3.10 (I have not tested above this.).
‼️ Git
‼️ Git Large File Storage.
‼️ Pandoc (only if you want to process.rtffiles).
🪟WINDOWS INSTRUCTIONS🪟
🟢 Nvidia GPU ➜ Install CUDA 11.8 or CUDA 12.1.
🔴 AMD GPU - Unfortunately, PyTorch does not currently support AMD GPUs on Windows. It's only supported on Linux. There are several ways to possibly get around this limitation, but I'm unable to verify since I don't have an AMD GPU. See HERE, HERE, and possibly HERE.
Download the ZIP file from the latest "release," unzip anywhere on your computer, and go into the src folder.
Within the src folder, open a command prompt and create a virtual environment:
python -m venv .
Activate the virtual environment:
.\Scripts\activate
python setup.py
And just follow the instructions.
Run this script if you want to doublecheck that you installed the Pytorch and gpu-acceleration software correctly:
python check_gpu.py
🐧LINUX INSTRUCTIONS🐧
🟢 Nvidia GPUs ➜ Install CUDA 11.8
🔴 AMD GPUs ➜ Install ROCm version 5.6.
THIS REPO might also help if AMD's instructions aren't clear.
Download the ZIP file from the latest "release," unzip anywhere on your computer, and go into the src folder.
Within the src folder, open a terminal window and create a virtual environment:
python -m venv .
Activate the virtual environment:
source bin/activate
python -m pip install --upgrade pip
🟢 Nvidia GPU:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
🔴 AMD GPU:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm5.6
🔵 CPU only:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
sudo apt-get install portaudio19-dev
sudo apt-get install python3-dev
pip install -r requirements.txt
Run this script if you want to doublecheck that you installed the Pytorch and gpu-acceleration software correctly:
python check_gpu.py
🍎APPLE INSTRUCTIONS🍎
All Macs with MacOS 12.3+ come with 🔘 Metal/MPS, which is Apple's implementation of gpu-acceleration (like CUDA for Nvidia and ROCm for AMD). I'm not sure if it's possible to install on an older MacOS since I don't have an Apple.
Install Xcode Command Line Tools.
Download the ZIP file from the latest "release," unzip anywhere on your computer, and go into the src folder.
Within the src folder, open a terminal window and create a virtual environment:
python -m venv .
Activate the virtual environment:
source bin/activate
python -m pip install --upgrade pip
pip install torch torchvision torchaudio
brew install portaudio
pip install -r requirements.txt
Run this script if you want to doublecheck that you installed the Pytorch and gpu-acceleration software correctly:
python check_gpu.py
My program allows you to transcribe a question into the clipboard and paste it into LM Studio. It uses the powerful Ctranslate2 library and the state-of-the-art "Whisper" models.
| Acceleration Support | Requirements | |
|---|---|---|
| Intel CPU | ✅ | |
| AMD CPU | ✅ | |
| Nvidia GPU | ✅ | CUDA |
| AMD GPU | ❌ | Will default to CPU |
| Apple CPU | ✅ | |
| Apple Metal/MPS | ❌ | Will default to CPU |
After following the installation instructions above, you can check which quantized versions of the Whisper models your CPU and GPU support. On Windows, use ctranslate2_compatibility.exe and on Linux or MacOS follow the instructions HERE.
As of Release v2.5, however, this is no longer mandatory because the program only displays compatible quantizations to choose from.
🔥USAGE INSTRUCTIONS🔥
Make sure you are in theh src folder, have opened a command prompt/terminal, and activated the virtual environment (see installation instructions).
python gui.py
Only systems running Windows with an Nvidia GPU will display metrics in the GUI. Feel free to request that I add AMD or Apple support.
The download embedding model button lets you choose to download multiple embedding models. The command prompt/terminal will state when the download is complete and unpacked. Don't attempt to create the vector database before it's done.
The set model directory button allows you to choose which embedding model to create/query the vector database. You can choose any of the embedding models you previously downloaded by selecting the folder in which the model files were downloaded to.
The choose documents button allows you to select which documents you want in the database. Symbolic links to the files are put within the "Docs_for_DB" folder, not the actual files, but you can manually drag and drop files there as well if you want. Feel free to select multiple files or click the add files button multiple times. To delete any files you have to manually delete them from the "Docs_for_DB" folder, however.
Remember, anytime you add/remove files you must recreate the vector database.
The file types that are supported are .pdf, .docx, .txt, .json, .enex, .eml, .msg, .csv, .xls, .xlsx, .rtf, .odt.
‼️ However, PDF files must have had OCR done on them.
As of release 2.6.1, you can also transcribe audio files for the database. The transcription process will automatically create a .txt file within the "Docs_for_DB" folder.
Remember, anytime you add/remove documents you must re-create the database.
The create database button...wait for it...creates the vector database! The command prompt will state when it's been "persisted." You should only conduct a search after you see this message.
- Before searching, open LM Studio and load a model.
‼️ Remember, only models that use the Llama-2 prompt format are supported by default. You can change the "prefix" or "suffix" to test out other models, but for 99% of use cases a basic model that uses the Llama-2 prompt format is sufficient. Mistral models use the formate and are excellent.
- ...then click the server tab on the left side.
- ...then click "Start Server" in the server tab.
Now type/transcribe your question and click "Submit Questions." The vector database will be queried. Your question along with any "contexts" from the database will be sent to the LLM within LM Studio for answer!
All suggestions (positive and negative) are welcome. "bbc@chintellalaw.com" or feel free to message me on the LM Studio Discord Server.
