Prompt for multiple choice question with quote extraction

[gdeleva]

Hello,

this is not an issue per se but a question how to best use Flan models.

Is there a way to design the prompt template in such a way that the model is expected to choose an option and at the same time a quote from the context that supports the choice?

For example, currently the following prompt

I have a big family. We all live in London. However, I was born in Rome.

Answer the following question by taking a quote from the text above.

Where do I live?
A. Rome
B. Berlin
C. London

is completed by "C.", but ideally it would be completed by "C. We all live in London.".

Best regards,

gdeleva

[shayne-longpre]

Hi @gdeleva. You may want to look at the prompt templates for the chain-of-thought datasets we trained on. E.g. here is a list for strategyQA. By prompting in the model in this way, it should make more of an attempt to answer with some justification, though I can't be sure it will quote the context directly.

Play around with the prompt to ask it to quote the context for the answer, in a chain-of-thought setting. If it isn't very good, you may need to finetune Flan-T5 models further with data in that format.

[gdeleva]

@shayne-longpre Thanks a lot for your advice.