This is the repo for ICLR 2024 paper: Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoning.
Full descriptions about this repo will come soon.
For any dataset D, Change the model_type in consistency_{D}.py to a specific model, and change sc_cum to the first sampling window size w. After running, you will get probs{w}.json under {model_type}_result/{D}/.
step 2. Obtain the expected sampling times and impact on performance under the specified sampling window size and maximum sampling times.
Set sc_num as the sampling window size and max_up as the maximum sampling times in control_scheme.py, and directly run it, you will get results like:
