Dose the pre-training data also use this prompt structure related to downstream tasks？

[yourName]

I read the gpt2 paper, but not sure whether the pre-training data from WebText will add format information.
For example, we konw data format will be english sentence = french sentencein the translation task. So during pre-training time, will we add similar promt to the training data?

Thanks!

[joan126]

interested about this