This is an NVIDIA AI Workbench example Project that demonstrates how to fine-tune a Mistral 7B large language model on a custom code instructions dataset using the QLoRA PEFT method. Users in the AI Workbench Beta Program can get up and running with this Project in minutes. Please note the project requirements:
- Operating System: Ubuntu 22.04, Windows (WSL2), MacOS 12+
- CPU requirements: None, (tested with Intel® Xeon® Platinum 8380 CPU @ 2.30GHz)
- GPU requirements: See Below
| GPU VRAM | Example Hardware | Compatible? | Additional Notes |
|---|---|---|---|
| 24 GB | RTX 3090/4090, RTX A5000/5500, A10/30 | Y | slow on 8-bit and full precision |
| 32 GB | RTX 5000 Ada, | Y | N/A |
| 40 GB | A100-40GB | Y | N/A |
| 48 GB | RTX 6000 Ada, L40/L40S, A40 | Y | N/A |
| 80 GB | A100-80GB | Y | N/A |
| >80 GB | 8x A100-80GB | Y | N/A |
Mistral 7B is a recent open-source language model developed by MistralAI that consistently delivers state-of-the-art results across a variety of natural language understanding and generation benchmarks. While this model serves as a strong baseline for multiple downstream tasks, it can lack in domain-specific knowledge or proprietary or otherwise sensitive information. Fine-tuning is often used as a means to update a model for a specific task or tasks to better respond to domain-specific prompts. This notebook walks through downloading the Mistral 7B model from Hugging Face, preparing a custom dataset on coding-related tasks and instructions, and using Quantized Low Rank Adaptation (QLoRA) to fine-tune the base model against the dataset. While we focus on a coding-specific task in this example, this methodology can be applied seamlessly to other tasks and data as well.
Have questions? Please direct any issues, fixes, suggestions, and discussion on this project to the DevZone Members Only Forum thread here.
If you have NVIDIA AI Workbench already installed, you can open this Project in AI Workbench on your choice of machine by:
If you do not NVIDIA AI Workbench installed, first follow the installation instructions for AI Workbench here. Then,
-
Fork this Project to your own GitHub namespace and copy the link
https://github.com/[your_namespace]/<project_name> -
Open NVIDIA AI Workbench. Select a location to work in.
-
Clone this Project onto your desired machine by selecting Clone Project and providing the GitHub link.
-
Wait for the project to build. You can expand the bottom Building indicator to view real-time build logs.
-
When the build completes, set the following configurations.
-
Environment→Mounts→Configure. Specify the file path of the mount, eg. where the fine-tuned Mistral model will live on your host machine.eg. if you would like your model directory to reside on your home path, enter
/home/[user]
-
-
On the top right of the window, select Jupyterlab.
-
Navigate to the code directory of the project. Then, open the fine-tuning notebook and get started. Happy coding!
Get started in the CLI by:
-
Forking this Project to your own GitHub namespace and copying the link
https://github.com/[your_namespace]/<project_name> -
Opening a shell and activating the Context you want to clone into by
$ nvwb list contexts $ nvwb activate <desired_context> -
Cloning this Project onto your desired machine by running
$ nvwb clone project <your_project_link> -
Opening the Project by
$ nvwb list projects $ nvwb open <project_name> -
Starting JupyterLab by
$ nvwb start jupyterlab- You may need to specify the file path of the mount, eg. where the fine-tuned Mistral model will live on your host machine, eg. if you would like your model directory to reside on the home path of the host system, enter
/home/[user]
- You may need to specify the file path of the mount, eg. where the fine-tuned Mistral model will live on your host machine, eg. if you would like your model directory to reside on the home path of the host system, enter
-
Navigate to the
codedirectory of the project. Then, open your Mistral fine-tuning notebook of choice and get started. Happy coding!
Tip: Use nvwb help to see a full list of NVIDIA AI Workbench commands.
This notebook has been tested with an NVIDIA A100-80gb GPU and the following version of NVIDIA AI Workbench: nvwb 0.21.3 (internal; linux; amd64; go1.21.3; Tue Mar 5 03:55:43 UTC 2024)
This NVIDIA AI Workbench example project is under the Apache 2.0 License
This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use. Third party components used as part of this project are subject to their separate legal notices or terms that accompany the components. You are responsible for confirming compliance with third-party component license terms and requirements.