composable-models / llm_multiagent_debate

ICML 2024: Improving Factuality and Reasoning in Language Models through Multiagent Debate

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Question about the Design of the `i`th Agent's Context at `t`th Round

BBQGOD opened this issue · comments

I would like to thank you for the great work you've put into this project.

I've been exploring the codebase and I came across a particular design choice that intrigued me. I noticed that the context for the ith agent at the tth round is determined by the replies from agents 0-(i-1) at the tth round, along with the replies from agents (i+1)-(n-1) at the t-1th round.

I was under the impression that a more common approach might be to use replies from agents 0-(i-1)&(i+1)-(n-1) at the t-1th round. Could you possibly explain the rationale/intuition behind this design?

Hi,

Thanks for your interest in the project. Yes the latter of using replies from agents 0-(i-1)&(i+1)-(n-1) at the t-1 round is more natural -- the newest version of the repo should be using that design choice.

Using the responses from 0-(i-1) at the tth round, along with the replies from agents (i+1)-(n-1) at the t-1th round was an unintended design choice in our original codebase (this design choice does encourage a more "debate-like behavior" where each agent responds to the new responses from other agents and might be an interesting direction of further exploration) -- I ran additional experiments and found that just using responses from all agents at round t-1 led to better results.