Prompt engineering is kind of like alchemy. There's no clear way to predict what will work best. It's all about experimenting until you find the right prompt. gpt-prompt-engineer
is a tool that takes this experimentation to a whole new level.
Simply input a description of your task and some test cases, and the system will generate, test, and rank a multitude of prompts to find the ones that perform the best.
-
Prompt Generation: Using GPT-4 and GPT-3.5-Turbo,
gpt-prompt-engineer
can generate a variety of possible prompts based on a provided use-case and test cases. -
Prompt Testing: The real magic happens after the generation. The system tests each prompt against all the test cases, comparing their performance and ranking them using an ELO rating system.
![Screen Shot 2023-07-04 at 11 41 54 AM](https://private-user-images.githubusercontent.com/41550495/250947004-f8171cff-1703-40ca-b9fd-f0aa24d07110.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM4NTgxNDksIm5iZiI6MTcyMzg1Nzg0OSwicGF0aCI6Ii80MTU1MDQ5NS8yNTA5NDcwMDQtZjgxNzFjZmYtMTcwMy00MGNhLWI5ZmQtZjBhYTI0ZDA3MTEwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE3VDAxMjQwOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTZjNDNjYTBlMjhkNjI3ZjZlMGZhOWVlYTU5MzUyNTQxNGVkZmI3NzFhMDdkNTk2ZDU4MTI3ZTVkZTEyZTdiZjAmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.fQNyfz_7lZ5dWRjSuuwZU2_3v0gwNeh_I-QKSlM0C6s)
-
ELO Rating System: Each prompt starts with an ELO rating of 1200. As they compete against each other in generating responses to the test cases, their ELO ratings change based on their performance. This way, you can easily see which prompts are the most effective.
-
Classification Version: The
gpt-prompt-engineer -- Classification Version
notebook is designed to handle classification tasks. It evaluates the correctness of a test case by matching it to the expected output ('true' or 'false') and provides a table with scores for each prompt.
![Screen Shot 2023-07-10 at 5 22 24 PM](https://private-user-images.githubusercontent.com/41550495/252494291-d5c9f2a8-97fa-445d-9c38-dec744f77854.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM4NTgxNDksIm5iZiI6MTcyMzg1Nzg0OSwicGF0aCI6Ii80MTU1MDQ5NS8yNTI0OTQyOTEtZDVjOWYyYTgtOTdmYS00NDVkLTljMzgtZGVjNzQ0Zjc3ODU0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE3VDAxMjQwOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTE0M2U2ZDU4YjQyODBkMWIyZWYzMDg3MDcyZmRhMTUxMjc2NmFjNTU3YWVjMTk4M2IwYTIxZjk2YmRmODUwZjMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.EUmbl4G3YQgM7L2frrVP4DG5--JvpAPBaixPuvgEiF8)
-
Weights & Biases Logging: Optional logging to Weights & Biases of your configs such as temperature and max tokens, the system and user prompts for each part, the test cases used and the final ranked ELO rating for each candidate prompt. Set
use_wandb
toTrue
to use. -
Portkey: Optional tool to log and trace all the prompt chains and their responses. Set
use_portkey
toTrue
to use.
-
Open the notebook in Google Colab or in a local Jupyter notebook. For classification, use this one.
-
Add your OpenAI API key to the line
openai.api_key = "ADD YOUR KEY HERE"
. -
If you have GPT-4 access, you're ready to move on. If not, change
CANDIDATE_MODEL='gpt-4'
toCANDIDATE_MODEL='gpt-3.5-turbo'
. If you're using the classification version, and don't have GPT-4 access, changemodel='gpt-4'
in the second cell tomodel='gpt-3.5-turbo'
.
- Define your use-case and test cases. The use-case is a description of what you want the AI to do. Test cases are specific prompts that you would like the AI to respond to. For example:
description = "Given a prompt, generate a landing page headline." # this style of description tends to work well
test_cases = [
{
'prompt': 'Promoting an innovative new fitness app, Smartly',
},
{
'prompt': 'Why a vegan diet is beneficial for your health',
},
{
'prompt': 'Introducing a new online course on digital marketing',
},
{
'prompt': 'Launching a new line of eco-friendly clothing',
},
{
'prompt': 'Promoting a new travel blog focusing on budget travel',
},
{
'prompt': 'Advertising a new software for efficient project management',
},
{
'prompt': 'Introducing a new book on mastering Python programming',
},
{
'prompt': 'Promoting a new online platform for learning languages',
},
{
'prompt': 'Advertising a new service for personalized meal plans',
},
{
'prompt': 'Launching a new app for mental health and mindfulness',
}
]
For the classification version, your test cases should be in the format:
test_cases = [
{
'prompt': 'I had a great day!',
'output': 'true'
},
{
'prompt': 'I am feeling gloomy.',
'output': 'false'
},
// add more test cases here
]
-
Choose how many prompts to generate. Keep in mind, this can get expensive if you generate many prompts. 10 is a good starting point.
-
Call
generate_optimal_prompt(description, test_cases, number_of_prompts)
to generate a list of potential prompts, and test and rate their performance. For the classification version, just run the last cell. -
The final ELO ratings will be printed in a table, sorted in descending order. The higher the rating, the better the prompt.
![Screen Shot 2023-07-04 at 11 48 45 AM](https://private-user-images.githubusercontent.com/41550495/250948262-324f90b8-c0ee-45fd-b219-6c44d9aa281b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM4NTgxNDksIm5iZiI6MTcyMzg1Nzg0OSwicGF0aCI6Ii80MTU1MDQ5NS8yNTA5NDgyNjItMzI0ZjkwYjgtYzBlZS00NWZkLWIyMTktNmM0NGQ5YWEyODFiLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE3VDAxMjQwOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTRkZmM0MDkxNmYwY2M3ZDIxOGNhMTMxNDcwOGM1MTMyNTA1MGYxNjg4ZTM0MGYzNTAzMjEyNGIzOWU3NjUzOWQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.yws6C-QdC1Fs2lMfRYA-6n_nJkKyzXVFIkT2qMkh_P0)
For the classification version, the scores for each prompt will be printed in a table (see the image above).
- have a number of different system prompt generators that create different styles of prompts, to cover more ground (ex. examples, verbose, short, markdown, etc.)
- automatically generate the test cases
- expand the classification version to support more than two classes using tiktoken
This project is MIT licensed.
Matt Shumer - @mattshumer_
Project Link: https://github.com/mshumer/gpt-prompt-engineer
Lastly, if you want to try something even cooler than this, sign up for Personal Assistant (most of my time is spent on this). It's basically an AI that can operate your web browser to complete tasks for you.