[falcon] requires trust_remote_code=True
MichaelSp opened this issue · comments
I tried running the awesome https://huggingface.co/tiiuae/falcon-7b-instruct via Docker and received this error message:
Model Error
Error parsing response from local inference: Traceback (most recent call last): File "/web/server/lib/inference/__init__.py", line 160, in __error_handler__ inference_fn(provider_details, inference_request) File "/web/server/lib/inference/__init__.py", line 572, in __local_text_generation__ hf = HFInference(inference_request.model_name) File "/web/server/lib/inference/huggingface/hf.py", line 29, in __init__ self.model, self.tokenizer = self.load_model(model_name) File "/web/server/lib/inference/huggingface/hf.py", line 38, in load_model config = AutoConfig.from_pretrained(model_name) # load config for model File "/opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 899, in from_pretrained raise ValueError( ValueError: Loading tiiuae/falcon-40b-instruct requires you to execute the configuration file in that repo on your local machine. Make sure you have read the code there to avoid malicious use, then set the option `trust_remote_code=True` to remove this error.
I struggle to find the place where to add the trust_remote_code=True
. Can you please help me?
You should be able to add the flag here https://github.com/nat/openplayground/blob/7f3f79035b3229aa471a9746d9af59035d5e71aa/server/lib/inference/huggingface/hf.py#LL38C13-L38C13
and that should allow you to load and infer no problem
Thank you, I can confirm that this worked.
config = AutoConfig.from_pretrained(model_name, trust_remote_code=True) # load config for model
if config.architectures:
model_classname = config.architectures[0]
model_class = getattr(MODULE, model_classname) # get model class from config
model = model_class.from_pretrained(model_name, config=config, trust_remote_code=True) # dynamically load right model class for text generation
else:
model = AutoModelForCausalLM.from_pretrained(model_name, device_map='auto' if DEVICE == 'cuda' else None, trust_remote_code=True)
now waiting for ggerganov/llama.cpp#1602
mosaicml/mpt-7b-instruct
needs the same extra trust_remote_code=True
as well, so this might be a more general thing?