Can't benchmark on medqa
JulienVig opened this issue · comments
In the evaluation folder, running python inference.py --checkpoint mistral --checkpoint_name mistral --benchmark medqa
fails with the error datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset
.
I managed to fix the error by setting the MedQA benchmark attribute self.subsets = ['med_qa_en_source]
By the way, the repository requirements.txt is missing the packages wandb, scikit-learn and openai (unused though), to be able to run the benchmark suite.
Thanks!