AssertionError: data parallel group is not initialized

Question

AssertionError: data parallel group is not initialized

victorup opened this issue a year ago · comments

Hi,
I encounter an error as follows:

Traceback (most recent call last):
File "/data33/private/xinpeng/codebase/CogView2/pretrain_coglm.py", line 244, in
training_main(args, model_cls=BaseModel, forward_step_function=forward_step, create_dataset_function=create_dataset_function)
File "/home/xinpeng/miniconda3/envs/cogview/lib/python3.9/site-packages/SwissArmyTransformer/training/deepspeed_training.py", line 66, in training_main
train_data, val_data, test_data = make_loaders(args, hooks['create_dataset_function'])
File "/home/xinpeng/miniconda3/envs/cogview/lib/python3.9/site-packages/SwissArmyTransformer/data_utils/configure_data.py", line 166, in make_loaders
group=mpu.get_data_parallel_group())
File "/home/xinpeng/miniconda3/envs/cogview/lib/python3.9/site-packages/SwissArmyTransformer/mpu/initialize.py", line 97, in get_data_parallel_group
assert _DATA_PARALLEL_GROUP is not None,
AssertionError: data parallel group is not initialized

Sleepy_chord · Answer 1 · Thu Apr 20 2023 22:55:29 GMT+0800 (China Standard Time)

this is strange. Have you made any changes on the file?
Please first try to install SwissArmyTransformer<0.3. If the error continues to appear, please tell me.

Sleepy_chord · Answer 2 · Fri Apr 21 2023 00:35:21 GMT+0800 (China Standard Time)

I think you should downgrade your deepspeed（it's 0.6.x in my memory）to and use the original code. I will add support for new deepspeed version soon发自我的手机-------- 原始邮件 --------发件人： Xinpeng Wang ***@***.***>日期： 2023年4月20日周四半夜11:51收件人： THUDM/SwissArmyTransformer ***@***.***>抄送： Sleepy_chord ***@***.***>, Comment ***@***.***>主题： Re: [THUDM/SwissArmyTransformer] AssertionError: data parallel group is not initialized (Issue #82) Thank you for your reply. This issue is a little complex. I have installed SwissArmyTransformer version SwissArmyTransformer==0.2.12 and attempted to fine-tune Cogview2 using deepspeed. The process I followed is shown in the following image: Could you please help me resolve these issues? And please let me know if there are any errors in my commands. Thanks! —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

Xinpeng Wang · Answer 3 · Fri Apr 21 2023 13:52:45 GMT+0800 (China Standard Time)

When I downgraded deepspeed to version 0.6.3, it worked. Thank you!