the logic of NO retrieval in long form inference
fate-ubw opened this issue · comments
excellent work ! I have a question about the logic of no retrieval. In multiple rounds of retrieval, if [No retrieval] appears for the first time, does it mean that all retrievals have ended, or does it mean that the second and third rounds of retrieval can still be carried out?
My understanding based on your paper and code is that if [No retrieval] appears for the first time, it means that all answers are generated without multiple retrievals.
waitting for your detailed explaination～
@AkariAsai
This problem has troubled me for a long time, really need you help. Thks for your reply～
Hi, sorry for my late response! I was busy with a paper release I was co-authoring in the past two weeks.
Yes, if Self-RAG generates [No retrieval] at the very beginning, the input query often doesn't require any retrieval through the generations (e.g., commonsense question, general natural language understanding) and thus we skip retrieval.
I'm glad to receive your response I still have two questions that I'm very confused about. hope for your reply~
You are mentioned that if the self-rag module predicts "no retrieval" in the first round, it won't perform additional rounds of retrieval. Following this logic, if the first two rounds are retrievals and the third round yields "no retrieval," it means that the final answer will be provided after the third round of retrieval. Is my understanding correct?
Based on your paper and your previous response, it seems that self-rag doesn't perform any further retrievals once it predicts "no retrieval" regardless of the round. However, in the provided training data (train.json) for the self-rag generator, I noticed that after [No Retrieval] there are instances where [Retrieval] appears.
{"instruction": "Problem: Write a positive yelp review.\nA: Awesome smoke shop and five stars because it's close to my house. The guys that work there are so friendly and helpful and they have a huge selection of literally everything. It's really small, located right next to Dukes Sports Bar.\n\nProblem: Write a negative yelp review.\nA: I used to visit this buffet pretty regularly because they HAD a keno board which there are only a few left on the strip. Now that it's gone there is no reason at all to eat here. The selection is eh, the food is eh and the price is the same as some of the better buffets on the strip. I will say that the service is pretty good. Unless you have a coupon for a free or very discounted buffet I would probably skip this one.\n\nProblem: Write a positive yelp review.\nA:", "output": "Some excellent sandwiches.[No Retrieval]The hot pastrami was sliced so thin that it was like butter.[No Retrieval]Not the biggest serving size, but one of the tastiest hot pastramis I ever had. [No Retrieval]I also had the classic Italian.Very nice! [Retrieval]<paragraph>Strip House\nmignon, bone-in rib eye, center cut chateaubriand and a classic porterhouse. Strip House is known for its minimal seasoning. Time Out New York says, “Order the New York strip and you’ll experience the sublime combination of a perfectly charred outside with a luscious rare-red inside.” In addition to steaks, the restaurants also serve, lobster, fish, veal, and rack of lamb. Side dishes range from the signature black truffle creamed spinach, French fries with herbs and garlic, crisp goose-fat potatoes Strip House also offers a dessert menu featuring favorites like the signature 24 layer chocolate cake. Strip House offers an extensive</paragraph>[Irrelevant]Only downside is their chili is kind of like Wendy's.[No Retrieval]Before purchasing, I asked the order taker how it was and he said that it is real chili with beans and not hot dog chili.[No Retrieval]That should have sounded the alarm, everyone knows real chili has no beans![Utility:5]", "input": "", "id": "flan_v2_31050", "dataset_name": "flan_v2"}
Could you explain why [Retrieval] occurs after [No Retrieval] in this case? This inconsistency has been confusing me for a while, and I would greatly appreciate your further explaination. Your jobs is really masterpiece~
Hi, sorry for my late response! I was busy with a paper release I was co-authoring in the past two weeks. Yes, if Self-RAG generates [No retrieval] at the very beginning, the input query often doesn't require any retrieval through the generations (e.g., commonsense question, general natural language understanding) and thus we skip retrieval.
Hi @AkariAsai,
Thanks for your reply. I have a related question if you could clarify please:
When generating the very first sentence, does Self-RAG generate [Retrieval] or [No Retrieval] token first before generating the first sentence? I'm asking because the output does not have the retrieval token before the first sentence, however, according to my understanding, it should always generate the retrieval token before predicting the next sentence?
For example (taken from Table 6 and 7):
Input: What genre is A Question and Answer Guide to Astronomy?
Output: (no [No Retrieval] token here) A Question and Answer Guide to Astronomy is a non-fiction book written by Patrick Moore and published in 1978.[Retrieval]<paragraph>[Relevant]The genre of this book is Astronomy.[No support / Contradictory][Utility:5]</s>
Input: When did sunday became the seventh day of the week
Output: (no [No Retrieval] token here) It is not clear when exactly Sunday became the seventh day of the week.[Retrieval]<paragraph>[Relevant]However, it is generally accepted that the current seven-day week was introduced by the ancient Sumerians, who divided their year into ten fixed months of 30 days each.[No support / Contradictory][No Retrieval]The seven-day week was then adopted by many other cultures, including the ancient Greeks and Romans.[No Retrieval]It is possible that the seven-day week was introduced in the Middle Ages, but
(Taken from your example in README):
Model prediction: (no [No Retrieval] token here) Twitter, Instagram, and WhatsApp are all social media platforms. [No Retrieval]WhatsApp is the odd one out because it is a messaging app, while Twitter and # Instagram are primarily used for sharing photos and videos.[Utility:5]</s>
Model prediction: (no [No Retrieval] token here) Sure![Retrieval]<paragraph><paragraph>
Does it just omit the first retrieval token?
Thank you very much!