Request for the correction of Santali script name

Question

Request for the correction of Santali script name

Prasanta-Hembram opened this issue 2 years ago · comments

Hi, I have downloaded FLORES-200 from here, When I reviewed the data for Santali language where the file name was written as sat_Beng.devtest (where the script code represents Santali in Bengali script) in both dev and devtest folder, but when viewed the file all contents were in Ol Chiki script (sat_Olck). Hence, I request for the change in file name from sat_Beng.devtest to sat_Olck.devtest.

Prasanta Hembram · Answer 1 · Tue Jul 12 2022 14:31:26 GMT+0800 (China Standard Time)

I fear that all datasets are having the same script name error for Santali language.

Angela Fan · Answer 2 · Tue Jul 12 2022 20:45:54 GMT+0800 (China Standard Time)

Thanks, investigating.

Angela Fan · Answer 3 · Thu Jul 14 2022 10:08:12 GMT+0800 (China Standard Time)

hello, we have confirmed that this is indeed a mislabeling. Our language identification, machine translation model, training data, evaluation data, and toxicity lists are in the OlChiki script of Santali. We will be updating the labels shortly and also update our writeup to reflect this.

Note that for the toxicity lists, the current one available is in Santali in Bengali script. We will be adding a list for Santali in OlChiki.

thank you so much for your feedback!