accuracy always comes 0, might be a bug in my code i am unable to find.
AbhinavJangra29 opened this issue · comments
❗BEFORE YOU BEGIN❗
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt
Describe the bug
A clear and concise description of what the bug is.
i want to perform a mmlu benchmark of a opensource model: https://huggingface.co/Telugu-LLM-Labs/Indic-gemma-2b-finetuned-sft-Navarasa-2.0
input format:
Instruction: {instruction}
Input: {input}
Response: {response}
To Reproduce
Steps to reproduce the behavior:
here is the notebook: https://github.com/AbhinavJangra29/Abhinav-Jangra/blob/main/MMLU_navarasa_issue.ipynb
- Go to '...'
- Click on '....'
- Scroll down to '....'
- See error
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]
Smartphone (please complete the following information):
- Device: [e.g. iPhone6]
- OS: [e.g. iOS8.1]
- Browser [e.g. stock browser, safari]
- Version [e.g. 22]
Additional context
Add any other context about the problem here.