LLM AutoEval simplifies the process of evaluating LLMs using a convenient Colab notebook. This tool is ideal for developers who aim to assess the performance of LLMs quickly and efficiently.
- Automated setup and execution using RunPod.
- Customizable evaluation parameters for tailored benchmarking.
- Summary generation and upload to GitHub Gist for easy sharing and reference.
Note: This project is in the early stages and primarily designed for personal use. Use it carefully and feel free to contribute.
- Benchmark suite:
nous: List of tasks: "agieval", "gpt4all", "truthfulqa", "bigbench" (popularized by Teknium and NousResearch). This is recommended.openllm: List of tasks: "arc", "hellaswag", "truthfulqa", "winogrande", "gsm8k" (like the Open LLM Leaderboard). It uses the vllm implementation to enhance speed (note that the results will not be identical to those obtained without using vllm). "mmlu" is currently missing because of a problem with vllm.
- Model: Enter the model id from Hugging Face.
- GPU: Select the GPU you want for evaluation (see prices here). I recommend using beefy GPUs (RTX 3090 or higher), especially for the Open LLM benchmark suite.
- Number of GPUs: Self-explanatory (not tested).
- Container disk: Size of the disk in GB.
- Debug: The pod will not be destroyed at the end of the run (not recommended).
Tokens use Colab's Secrets tab. Create two secrets called "runpod" and "github" and add the corresponding tokens you can find as follows:
- Runpod: Please consider using my referral link if you don't have an account yet. You can create your token here under "API keys" (read & write permission). You'll also need to transfer some money there to start a pod.
- GitHub: You can create your token here (read & write, can be restricted to "gist" only).
You can compare your results with:
- Models like OpenHermes-2.5-Mistral-7B, Nous-Hermes-2-SOLAR-10.7B, or Nous-Hermes-2-Yi-34B.
- Teknium stores his evaluations in his LLM-Benchmark-Logs.
You can compare your results with those listed on the Open LLM Leaderboard.
- "Error: File does not exist": This task didn't produce the JSON file that is parsed for the summary. Activate debug mode and rerun the evaluation to inspect the issue in the logs.
- "700 Killed" Error: The hardware is not powerful enough for the evaluation. This happens when you try to run the Open LLM benchmark suite on an RTX 3070 for example.
- Outdated CUDA Drivers: That's unlucky. You'll need to start a new pod in this case.
Special thanks to EleutherAI for the lm-evaluation-harness, dmahan93 for his fork that adds agieval to the lm-evaluation-harness, NousResearch and Teknium for the Nous benchmark suite, and vllm for the additional inference speed.