Large Language Models as Optimizers

This repository contains the code for the paper

Large Language Models as Optimizers
Chengrun Yang*, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, Xinyun Chen* [* Equal Contribution]
arXiv: 2309.03409

Dependency requirements

The code has been verified to work under Python 3.10.13 with the following dependencies:

- absl-py (2.0.0)
- google.generativeai (0.1.0)
- immutabledict (3.0.0)
- openai (0.27.2)

Usage

Prompt optimization

Use opro/optimization/optimize_instructions.py, follow the steps at the top.

A quickstarter:

python optimize_instructions.py --optimizer="gpt-3.5-turbo" --scorer="text-bison" --instruction_pos="Q_begin" --dataset="gsm8k" --task="train" --palm_api_key="<your_palm_api_key>" --openai_api_key="<your_openai_api_key>"

Prompt evaluation

Use opro/evaluation/evaluate_instructions.py, follow the steps at the top.

A quickstarter:

python evaluate_instructions.py --scorer="text-bison" --dataset="gsm8k" --task="test" --instruction_pos="Q_begin" --evaluate_training_fold=false --evaluate_test_fold=true --palm_api_key="<your_palm_api_key>"

Supported models

The code in this repository currently supports text-bison and GPT models. Alternatively, you may serve your own model and plug it in here, similar to the existing prompting APIs in opro/prompt_utils.py.

Precaution on API costs

Calling the PaLM or GPT APIs for prompt optimization and evaluation may incur unexpectedly large costs. Please carefully estimate the cost and/or start with lighter use (e.g., evaluate on a smaller portion of the benchmark dataset or run optimization for fewer steps) before the formal experimentations, or prompt self-served models instead.

Citation

If you have used our code in your research, please cite our paper:

@article{yang2023large,
  title={Large language models as optimizers},
  author={Yang, Chengrun and Wang, Xuezhi and Lu, Yifeng and Liu, Hanxiao and Le, Quoc V and Zhou, Denny and Chen, Xinyun},
  journal={arXiv preprint arXiv:2309.03409},
  year={2023}
}

Disclaimer: this is not an officially supported Google product.

google-deepmind / opro