google / seqio

Task-based datasets, preprocessing, and evaluation for sequence models.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

`tokenize_and_append_eos` needs another requrired input (`output_features`)

puraminy opened this issue · comments

commented

tokenize_and_append_eos needs another requrired input (output_features) how can I use this function as preprocessor, how to pass output features?

It's the way I tired to use it

preprocessors=[
          functools.partial(
              t5.data.preprocessors.parse_tsv,
              field_names=["input_text", "target_text"]),
          seqio.preprocessors.tokenize_and_append_eos,
    ],

output_features will automatically get passed in as part of the get_dataset pipeline.