Behavior in cases the template is not applicable
kasnerz opened this issue · comments
If the template is not applicable for a particular example, the prompt.apply() method returns only a single-element list with an empty string instead of a two-element list with the prompted input and the reference answer. This contradicts the docstring which says that two elements are returned.
Should we just update the docstring, or handle this case with more care?
Example: the e2e_nlg_cleaned dataset with QA-style templates such as food_qa (1acabbc3-c9b9-4624-a684-29faeccff46f). This manifests in the web interface by displaying "∅∅∅ Blank result" instead of an input-target pair.
hey @kasnerz !
yes that sounds about right. we should modify the docstring. Would you like to open a pr for this?