`tokenize_and_append_eos` needs another requrired input (`output_features`)
puraminy opened this issue · comments
tokenize_and_append_eos
needs another requrired input (output_features
) how can I use this function as preprocessor, how to pass output features?
It's the way I tired to use it
preprocessors=[
functools.partial(
t5.data.preprocessors.parse_tsv,
field_names=["input_text", "target_text"]),
seqio.preprocessors.tokenize_and_append_eos,
],
output_features
will automatically get passed in as part of the get_dataset
pipeline.