SP parameter device type error

Question

SP parameter device type error

dongsungkim opened this issue 2 years ago · comments

dongsungkim commented 2 years ago

How to reproduce

Environment

OS : CentOS 7.9
Python version : 3.7
Transformers version : 4.21.3
Whether to use Docker:
Misc.:

Description

model_no_sp = GPT2LMHeadModel(GPT2Config.from_pretrained(configs["model_name"])).cuda()
model_sp = GPT2LMHeadModel(GPT2Config.from_pretrained(configs["model_name"]))

model_sp = SequenceDataParallel(
model_sp,
parallel_context=parallel_context,
)

Error comes from init of _DistributedDataParallel due to parameters are on CPU not on GPU.

Need to remove device_type check code of parameter in _DistributedDataParallel