Error when evaluating TEXT_TO_TEXT_GENRATION

[Zijian Wang]

When running python evaluate.py -t ButterFingersPerturbation -task "TEXT_TO_TEXT_GENERATION" -p 1, there will be error of

Here is the performance of the model on the transformed set
Length of Evaluation dataset is 226
Traceback (most recent call last):
  File "evaluate.py", line 67, in <module>
    if_filter
  File "./NL-Augmenter/evaluation/evaluation_engine.py", line 41, in evaluate
    percentage_of_examples=percentage_of_examples,
  File "./NL-Augmenter/evaluation/evaluation_engine.py", line 115, in execute_model
    split=f"test[:{percentage_of_examples}%]",
  File "./NL-Augmenter/evaluation/evaluate_text_generation.py", line 44, in evaluate
    dataset, summarization_pipeline, transformation=operation
  File "./NL-Augmenter/evaluation/evaluate_text_generation.py", line 70, in transformation_performance
    pt_dataset, summarization_pipeline
  File "./NL-Augmenter/evaluation/evaluate_text_generation.py", line 81, in performance_on_dataset
    article, gold_summary = example
  File "./NL-Augmenter/dataset.py", line 301, in <genexpr>
    yield (datapoint[field] for field in self.fields)
TypeError: string indices must be integers

[Zhenhao Li]

Thanks for pointing this out!

It should be fixed in this PR #92. The bug was basically caused by this line. When applying SentenceOperation, the returned value should be a list of datapoint (though the length is only 1).

Feel free to comment if there're any other issues.