Giters

anchen1011 / FireAct

FireAct: Toward Language Agent Fine-tuning

Home Page:https://fireact-agent.github.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

How to make reflexion results on HotPotQA

pengjiao123 opened this issue · comments

commented

Good work！

I want to know how the author did reflexion results on hotpotqa.
I have read the original author's method. His evaluator uses the ground truth to the questions in hotpotqa, and then uses em as an evaluation. I would like to ask you how to get the result of reflexion （Especially the evaluator） when doing comparative experiments.

Thanks for your reply

commented

Anothor two question

image
Is Reflexion here generating samples using gpt-4 (we simply prompt for reflections at the 6th and 10th ReAct round?) and then fine-tuning the LLM ?

image

why are the temperatures different here? (Why not use 0 or 0.6 uniformly or do both experiments)

commented

Hi there, sorry for the delayed reply. We apply reflexion when a solution is not reached at certain step.

Is Reflexion here generating samples using gpt-4 (we simply prompt for reflections at the 6th and 10th ReAct round?) and then fine-tuning the LLM ?
Yes

why are the temperatures different here? (Why not use 0 or 0.6 uniformly or do both experiments)
Thanks for the suggestion. We will look into these details and see if we can experiment with both settings.