[transformers/slimpajama] where is lm_dataformat?

Question

[transformers/slimpajama] where is lm_dataformat?

tbarton16 opened this issue a year ago · comments

I am trying to deduplicate some data. When I run main.py I get an error ModuleNotFoundError: No module named 'lm_dataformat' And looking around the module, for the public version, lm_dataformat is not included. I would appreciate help with this.

bhargav-cerebras · Answer 1 · Wed Jun 14 2023 01:54:37 GMT+0800 (China Standard Time)

Hi @tbarton16 , thanks for catching this issue.. we'll update our requirements.txt. Meanwhile, did you try pip install lm_dataformat in your environment?

Theresa Barton · Answer 2 · Wed Jun 14 2023 02:17:13 GMT+0800 (China Standard Time)

I installed lm_dataformat. I can run python -c 'import lm_datformat'. Running main still produces ModuleNotFoundError: No module named 'lm_dataformat.lm_dataformat'

Theresa Barton · Answer 3 · Wed Jun 14 2023 02:21:04 GMT+0800 (China Standard Time)

Unless you have a different version the line should be from lm_dataformat import Reader not https://github.com/Cerebras/modelzoo/blob/main/modelzoo/transformers/data_processing/slimpajama/preprocessing/filter.py#L14

faisalak-cerebras · Answer 4 · Wed Jun 14 2023 07:00:35 GMT+0800 (China Standard Time)

Hi @tbarton16, can you run git clone git@github.com:leogao2/lm_dataformat.git in the directory https://github.com/Cerebras/modelzoo/tree/main/modelzoo/transformers/data_processing/slimpajama. It should resolve your issue.
Thanks.