π₯This work aims to explore the debating capability of LLMs by proposing the MAD framework, which stands for Multi-Agents Debate.
"Truth emerges from the clash of adverse ideas."
"ηηθΆθΎ©θΆζγ"
The cognitive behavior of large language models (LLMs) has garnered significant attention in recent times. For example, self-reflection, a concept that usually refers to the process of introspection and examination of a person's own thoughts, has also been demonstrated effective with LLMs in solving challenging NLP tasks. However, we point out that self-reflection can easily fall into the degeneration of thoughts (DoT) issue in the follow scenarios:
- Bias and Distorted Perception: Self-perception can be influenced by biases, preconceived notions, and distorted thinking patterns. If an individual's self-reflection is clouded by such biases or distorted thinking, it can lead to πinaccurate conclusions and hinder personal growth.
- Rigidity and Resistance to Change: Self-reflection often involves challenging one's beliefs, assumptions, and behaviors. If an individual is resistant to change or holds rigid beliefs, they may πstruggle to engage in meaningful self-reflection that leads to personal growth.
- Limited External Feedback: Self-reflection is primarily an internal process, but external feedback can provide valuable perspectives and insights. Without seeking or considering external feedback, an individual may πmiss important blind spots or alternative viewpoints that can enrich their self-reflection.
Figure 1: Comparison between debate and reflection.
In this project, we have embarked on a journey to explore the potential of a debating interaction framework among LLMs. With MAD, the nature of agents being in the state of 'tit for tat' determines that (1) the distorted thinking of one agent can be corrected by the other one π; (2) the resistance to change of one agent will be complemented by the other one π; and (3) either agent can provide external feedback for each other π.
Obviously, MAD is less likely to have the DoT issue and can exploit more potential of LLMs. Experiments show that MAD brings significant and consistent improvements on Counterintuitive QA and Commonsense-MT tasks.
JOIN US on this journey of exploring the interaction and debating capability with LLMs. πππ
Preparation
pip install -r code/requirements.txt- Set your openai API_KEY in
code/response-turbo.sh
Run MAD
sh code/response-turbo.sh
Table 1: Results on 10 Counterintuitive QA Examples.
When Alice walks up the hill, her speed is 1 m/s and when she goes down the hill, her speed is 3 m/s. Then when Alice walks up and down the hill, what is her average speed? (1.5m/s)
Figure 2: An Animation to Show the Process of MAD.
Debate process
:Alice's average speed can be calculated by adding her speed going up the hill and her speed going down the hill, and then dividing by 2. So, (1 m/s + 3 m/s) / 2 = 2 m/s. Therefore, Alice's average speed is 2 m/s.
: I disagree with your answer. The problem doesn't give us any information about the distance Alice covers while going up and down the hill. So, we cannot simply add her speeds and divide by 2 to get her average speed. We need to know the distance she covers to calculate her average speed.- : That's a good point. But we can assume that Alice covers the same distance going up and down the hill. In that case, we can use the formula for average speed, which is total distance divided by total time. If Alice covers a distance of d going up and down the hill, then her total distance is 2d. Her time going up the hill is d/1 = d, and her time going down the hill is d/3. So, her total time is d + d/3 = 4d/3. Therefore, her average speed is 2d/(4d/3) = 3/2 m/s.
Case 2
We have 1 ton apples which contain 90% water. After a few days, those apples only have 80% water. What is the weight of those apples now? (0.5ton)-
: First, we need to find out how much of the original weight was water. If we have 1 ton of apples with 90% water, then the weight of water in those apples is 0.9 x 1 ton = 0.9 tons.
Next, we need to find out how much of the new weight is water. If the apples now have 80% water, then the weight of water in those apples is 0.8 x new weight.We can set up an equation:
0.8 x new weight = 0.9 tons
To solve for the new weight, we can divide both sides by 0.8:
new weight = 0.9 tons / 0.8
new weight = 1.125 tons
Therefore, the weight of the apples now is 1.125 tons. -
: It's important to note that the weight of the apples has decreased due to the loss of water, but the weight of the remaining solids remains the same. This is why we need to calculate the weight of remaining solids in both scenarios to find the new weight of the apples.
-
: You are correct. The weight of the apples has decreased due to the loss of water, but the weight of the remaining solids remains the same. Therefore, we need to calculate the weight of the remaining solids in both scenarios to find the new weight of the apples.
We can set up an equation:
0.2 x new weight = 0.1 tons
To solve for the new weight, we can divide both sides by 0.2:
new weight = 0.1 tons / 0.2
new weight = 0.5 tons
Therefore, the weight of the apples now is 0.5 tons.
Table 2: Results on Commonsense MT.
Given the Chinese sentence "εζζδΊΊδΈδΈͺεΈγ", please provide its translation in English.
- : I believe the correct translation of the Chinese text "εζζδΊΊδΈδΈͺεΈ" is "Eat up an enemy division.β The word "εζ" literally means "eat up" or βconsume."
- : I disagree with your translation. I believe the correct translation of the Chinese text "εζζδΊΊδΈδΈͺεΈ" is "Eliminate an enemy division.β While "εζ" can be translated as "eat up" or "consume," it can also mean "eliminate" or "annihilate" in a military context. In this case, it is more appropriate to use "eliminate" as it conveys the idea of completely destroying the enemy division.
: Both sides presented valid arguments and translations that accurately convey the meaning of the original Chinese text. The choice between 'eat up' and 'eliminate' may depend on the context and tone of the text. Therefore, there is no clear preference for either side at this time.- : While "eliminate" does convey the idea of completely destroying the enemy division, it doesn't capture the connotation of consuming or devouring the enemy that is implied by βεζ."
- : In a military context, "εζ" is often used to describe the complete destruction of the enemy, rather than the consumption of their resources or territory.
- : Both sides presented valid arguments, but the negative side's argument that 'eliminate' better captures the intensity and seriousness of the military language used in the original Chinese text was more convincing. Additionally, 'eliminate' accurately conveys the idea of completely destroying the enemy division. Therefore, the correct English translation of the Chinese text 'εζζδΊΊδΈδΈͺεΈ' is 'Eliminate an enemy division.'
Case 2
Given the Chinese sentence "δ»δ»ει¨ζε°δΊδΈε°ει γ", please provide its translation in English.- : The phrase βδ»ει¨β literally translates to βfrom the back door.β
- : I actually disagree with your translation. I believe a more accurate translation would be "He got a lot of famous liquor from the back door. Lastly, while "δ»ει¨" does literally translate to "from the back door," it is a common Chinese idiom that means obtaining something through unofficial or illegitimate means. Therefore, the use of "from the back door" in the translation is not entirely accurate. Instead, the use of "from an unofficial source" or "through unofficial means" would be more appropriate.
- : Both sides presented valid arguments and translations that accurately capture the meaning of the original Chinese text. The choice between wine and liquor is a matter of interpretation and context. The use of from the back door versus from an unofficial source is a matter of preference and style. Therefore, there is no clear preference for a translation candidate.
- : I agree that "δ»ει¨" is a common Chinese idiom that means obtaining something through unofficial or illegitimate means. The use of "from an unofficial source" or "through unofficial means" can be more appropriate than "from the back door.β
- : Yes, I agree with your perspective.
- : While both translations accurately capture the meaning of the original Chinese text, the use of 'liquor' in candidate 2 is more accurate as 'ει
' is more commonly used to refer to liquor. Additionally, the use of 'from an unofficial source' in candidate 3 more accurately conveys the connotation of 'δ»ει¨' as obtaining something through unofficial or illegitimate means. Therefore, the correct translation is: 'He got a lot of famous liquor from an unofficial source.'
- 0-Shot CoT: Large Language Models are Zero-Shot Reasoners (NeurIPS 2022)
- Self-Consist: Self-Consistency Improves Chain of Thought Reasoning in Language Models (ICLR 2023)
- Self-Reflect: Reflexion: an autonomous agent with dynamic memory and self-reflection (arxiv 2023)
- MAPS: Exploring Human-Like Translation Strategy with Large Language Models (arxiv 2023)
@misc{liang2023mad,
author = {Tian Liang, Zhiwei He, Wenxiang Jiao, Yan Wang, Rui Wang, Yujiu Yang, Xing Wang, Zhaopeng Tu, Shuming Shi},
title = {Exploring Multi-Agents Debate with Large Language Models},
year = {2023},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {https://github.com/Skytliang/Multi-Agents-Debate},
}