Giters

symflower / eval-dev-quality

DevQualityEval: An evaluation benchmark 📈 and framework to compare and evolve the quality of code generation of LLMs.

Home Page:https://symflower.com/en/company/blog/2024/dev-quality-eval-v0.4.0-is-llama-3-better-than-gpt-4-for-generating-tests/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Evaluation run for all "good open weight models" with all available quantizations and different GPUs

zimmski opened this issue · comments

commented

See https://www.reddit.com/r/LocalLLaMA/comments/1dlsxab/comment/l9rzjj7/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

Not sure on how we should do that yet. CPU-only-inference will break us here, and speed-metrics are important as well.