[falcon] requires trust_remote_code=True

[Michael]

I tried running the awesome https://huggingface.co/tiiuae/falcon-7b-instruct via Docker and received this error message:

Model Error

Error parsing response from local inference: Traceback (most recent call last): File "/web/server/lib/inference/__init__.py", line 160, in __error_handler__ inference_fn(provider_details, inference_request) File "/web/server/lib/inference/__init__.py", line 572, in __local_text_generation__ hf = HFInference(inference_request.model_name) File "/web/server/lib/inference/huggingface/hf.py", line 29, in __init__ self.model, self.tokenizer = self.load_model(model_name) File "/web/server/lib/inference/huggingface/hf.py", line 38, in load_model config = AutoConfig.from_pretrained(model_name) # load config for model File "/opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 899, in from_pretrained raise ValueError( ValueError: Loading tiiuae/falcon-40b-instruct requires you to execute the configuration file in that repo on your local machine. Make sure you have read the code there to avoid malicious use, then set the option `trust_remote_code=True` to remove this error. 

I struggle to find the place where to add the trust_remote_code=True. Can you please help me?

[Zain Huda]

You should be able to add the flag here https://github.com/nat/openplayground/blob/7f3f79035b3229aa471a9746d9af59035d5e71aa/server/lib/inference/huggingface/hf.py#LL38C13-L38C13
and that should allow you to load and infer no problem

[Michael]

Thank you, I can confirm that this worked.

        config = AutoConfig.from_pretrained(model_name, trust_remote_code=True) # load config for model
        if config.architectures:
            model_classname = config.architectures[0]
            model_class = getattr(MODULE, model_classname) # get model class from config
            model = model_class.from_pretrained(model_name, config=config, trust_remote_code=True) # dynamically load right model class for text generation
        else:
            model = AutoModelForCausalLM.from_pretrained(model_name, device_map='auto' if DEVICE == 'cuda' else None, trust_remote_code=True)

[Michael]

now waiting for ggerganov/llama.cpp#1602

[Gergely Imreh]

mosaicml/mpt-7b-instruct needs the same extra trust_remote_code=True as well, so this might be a more general thing?