eval generation path issue
sia-cerebras opened this issue · comments
Hello, I'm trying to use your eval pipeline.
I ran ./inference_pipeline.sh -b pubmedqa -c gpt2 -s 0 -m 0 -out_dir out_dir
After it is done with generation I get the following error:
Stored pubmedqa generations to the following path: ../benchmarks/generations/pubmedqa-gpt2.jsonl
Traceback (most recent call last):
File "evaluate.py", line 475, in <module>
main(args)
File "evaluate.py", line 390, in main
data = load_jsonl(path)
File "evaluate.py", line 39, in load_jsonl
with open(filename, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '../benchmarks/generations/pubmedqa/pubmedqa-gpt2.jsonl'
It looks like the generations are saved in: meditron/benchmarks/generations/pubmedqa-gpt2.jsonl
But eval looks for them in: meditron/benchmarks/generations/pubmedqa/pubmedqa-gpt2.jsonl'