`RESPONSE` contains lot longer text than is expected based on the `output_features` and `max_sequence_length`.
amankhandelia opened this issue · comments
RESPONSE consumed by ROUGEScoreMetric function, gets a lot longer text than is expected based on the output_features and max_sequence_length. Even when the max_sequence_length is 8 or 16 tokens, it RESPONSE contains the text which is as long as the text in prompt_template.
Based on my investigation, it is happening because in get_decoded_targets_and_predictions condition is wrong, instead of targets != IGNORE_INDEX_TOKEN_ID, it is set to predictions[PREDICTIONS] != IGNORE_INDEX_TOKEN_ID. From what I understand we should be using the targets index to truncate the predictions correctly.
When I apply this change I get the correct metric value, matching up with the expectations given the results seen during finetuning
- Python version: 3.11
- Ludwig version: 0.10.1