How to inference GPT2 with DeepSpeed?
cdj0311 opened this issue · comments
Hi,
I only found bloom model inference in scripts, but how to inference GPT2 with deepspeed which trained with Megatron-DeepSpeed?
I convert megatron-deepspeed model to transformers with tools/convert_checkpoint/deepspeed_to_transformers.py, then modify checkpoint_json = {"type": "Megatron","checkpoints": file_list,"version": 1.0} to inference , but I get an error as follows:
/home/anaconda3/envs/py37/lib/python3.7/site-packages/deepspeed/module_inject/replace_module.py", line 789, in replace_transformer_layer
checkpoint = checkpoint_dict['checkpoints']
TypeError: 'MegatronSDLoader' object is not subscriptable
Hf models are readily compatible with deepspeed.
Deepspeed.initialize wraps any nn.module class object