EleutherAI / lm-evaluation-harness

A framework for few-shot evaluation of language models.

Home Page:https://www.eleuther.ai

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Single doc results of model inference not found

EugeneYCSong1212 opened this issue · comments

Hi~Referring to the doc, I think if I add '--output' and '--log_samples' argument I could get every inference result from one doc. I have executed the basic demo with these two arguments like this:lm_eval --model hf --model_args pretrained=EleutherAI/gpt-j-6B --tasks hellaswag --device cuda:0 --batch_size 8 --output_path /home/work/ais-harness-data-test/harness-res/0418/ --log_samples
I have found 'pretrained=EleutherAI__gpt-j-6B_hellaswag.jsonl' under the result output dir but I am confused of the result. The attachment is a result demo of one doc.
result_demo.txt
I have found the dataset doc details and some results after computation. However, I do not find the inference result of the model. Has it been provided? If not, is there any way to obtain it or will it be supported in the future. I am looking forward to your reply, thanks a lot