The official repo for the arxiv preprint ALGO: Synthesizing Algorithmic Programs with Generated Oracle Verifiers.
We built upon the following open-source projects to automatically prompt ChatGPT code interpreter, collect problem statements from LeetCode and Codeforces, and to submit solutions to LC and CF.
The prompts and the tags we used for generating the candidate solutions, the oracles, the data generators, the data verifiers are listed in prompts.py as format strings.
The results for CodeContests are in codecontests.zip.
For each problem there are three files. For instance, the files for problem 1575_A are 1575_A_bf.py, 1575_A_test.py and 1575_A_data.json.
-
1575_A_bf.pycontains the reference oracle for the problem, where the entry point is the functionsolution. -
1575_A_test.pycontains the data generator for the problem, where the entry point is the functionbatch_gen_inputs. -
1575_A_data.jsoncontains the metadata and example cases for the problem. Specifically, the problem definition is in thepromptfield, and the test cases generated by ALGO are listed in theexamples_separatefield.
We use the ALGO-generated test cases to sort the solutions generated by Codex (which are directly taken from the CodeT repo) according to the number of tests they pass. For each problem there are about ~1000 candidates. They are listed along with the number of test cases in sorted_codecontests.json. Note that they are sorted with up to 20 test cases only.
The 35 leetcode problems we collected for benchmarking ALGO is listed in leetcode_data/, where {problem_id}.md contains the problem statement and metadata.json contains the tags, public example cases and the hints (which we do not use).
The verifiers for LeetCode problems are in leetcode_results/. For each problem there is a {problem_id}_response.json, where the reference oracle is listed in the code_bruteforce field, the data generators are listed in the data_validator, data_generator, data_random_generator, data_tricky_generator fields.