Jack-Lingjie / IFeval

IFEval: Instruction Following Eval

This is not an officially supported Google product.

This repository contains source code and data for Instruction Following Evaluation for Large Language Models

Dependencies

Please make sure that all required python packages are installed via:

pip3 install -r requirements.txt

How to run

You need to create a jsonl file with two entries: prompt and response. Then, call evaluation_main from the parent folder of instruction_following_eval. For example:

# Content of `--input_response_data` should be like:
# {"prompt": "Write a 300+ word summary ...", "response": "PUT YOUR MODEL RESPONSE HERE"}
# {"prompt": "I am planning a trip to ...", "response": "PUT YOUR MODEL RESPONSE HERE"}
# ...
python3 -m instruction_following_eval.evaluation_main \
  --input_data=./instruction_following_eval/data/input_data.jsonl \
  --input_response_data=./instruction_following_eval/data/input_response_data_gpt4_20231107_145030.jsonl \
  --output_dir=./instruction_following_eval/data/

Reference

If you use our work, please consider citing our preprint:

@article{zhou2023instruction,
  title={Instruction-Following Evaluation for Large Language Models},
  author={Zhou, Jeffrey and Lu, Tianjian and Mishra, Swaroop and Brahma, Siddhartha and Basu, Sujoy and Luan, Yi and Zhou, Denny and Hou, Le},
  journal={arXiv preprint arXiv:2311.07911},
  year={2023}
}