LAMBDAPROMPT_BACKEND = StarCoder causes a runtime error
thinkORo opened this issue · comments
When I activate the local execution, I get the following error message:
ValueError: The current "device_map" had weights offloaded to the disk. Please provide an "offload_folder" for them. Alternatively, make sure you have "safetensors" installed if the model you are using offers the weights in this format.
To activate the local execution, I did the following:
os.environ['LAMBDAPROMPT_BACKEND'] = 'StarCoder'
os.environ['SKETCH_USE_REMOTE_LAMBDAPROMPT'] = 'False'
os.environ['HF_ACCESS_TOKEN'] = 'myToken'
Unfortunately, I cannot find a way to set the offload_folder.
How can I do that? Note: "safetensors" is already installed
I found a workaround as I didn't find the time to follow the complete execution chain fro Sketch.
I patched the responsive module, modeling_utils.py, and hard-coded the "offload_folder" for the required methods:
- _load_state_dict_into_meta_model()
- _load_pretrained_model()
But it would be nice not to patch the underlying modules.
Therefore, any ideas are welcome.