data preprocessing
Lj4040 opened this issue · comments
What do SOURCE and TARGER stand for in data preprocessing? Could you explain them? Thank you for your reply
As it was mentioned , source is original text, target is corrected text
For example, what I downloaded is the FCE data set, which contains M2 file and json file. In this file, there is no distinction between correct and incorrect sentences. How should I pass the data processing file.I would like to ask for your guidance, for which I greatly appreciate it
Only the downloaded synthetic data set has correct and incorrect sentences, do we have to use the synthetic data to pass in?
You can take a look at the M2scorer repository and specifically the edit_creator.py script.
To get original/source sentences, you can simply run cat myfile.m2 | grep "^S " | cut -c3- > myfile.src
Sincerely thank you for your answer, I will try
Dear author, after data processing, the data set has become the following picture. This format file is quite different from the M2 file, so I'm not sure if it's correct.May I ask if the data set in this format is correct?
Because I am a beginner of GEC, some questions may be a little naive. I hope you can understand. Thank you for your reply
Yes, this is a specific format for training to save only input tokens and corresponding tags