Extra hugging face model need to be downloaded.

[FangXianze]

In file download_hf_models.py, runwayml/stable-diffusion-v1-5 and lllyasviel/ControlNet-v1-1 is downloaded in advance. But when I ran the script train_lora.py, errors occured like below:

OSError: Can't load tokenizer for 'openai/clip-vit-large-patch14'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'openai/clip-vit-large-patch14' is the correct path to a directory containing all relevant files for a CLIPTokenizer tokenizer.

So I wonder if I need to download the model 'openai/clip-vit-large-patch14' and where should I put the file.
Thanks a lot for your excellent work and the code sharing!

[GaussianObject]

Thank you for your feedback and kind words!

The CLIP model within ControlNet is defined at:

GaussianObject/ldm/modules/encoders/modules.py

Lines 88 to 100 in c087d48

 class FrozenCLIPEmbedder(AbstractEncoder): 
     """Uses the CLIP transformer encoder for text (from huggingface)""" 
     LAYERS = [ 
         "last", 
         "pooled", 
         "hidden" 
     ] 
     def __init__(self, version="openai/clip-vit-large-patch14", device="cuda", max_length=77, 
                  freeze=True, layer="last", layer_idx=None):  # clip-vit-base-patch32 
         super().__init__() 
         assert layer in self.LAYERS 
         self.tokenizer = CLIPTokenizer.from_pretrained(version) 
         self.transformer = CLIPTextModel.from_pretrained(version) 

Therefore, the transformers module should automatically download the model weights into ~/.cache/huggingface/hub. It's actually unusual to encounter an OSError since the code is designed to access CLIP weights from the Hugging Face Hub directly.

To troubleshoot, please try running the following code:

from transformers import CLIPTokenizer, CLIPTextModel

version="openai/clip-vit-large-patch14"
tokenizer = CLIPTokenizer.from_pretrained(version)
transformer = CLIPTextModel.from_pretrained(version)

and check if any errors occur.

[FangXianze]

Thanks for your prompt reply! I solve the problem by using the mirror website of hugging face.
HF_ENDPOINT=https://hf-mirror.com python train_lora.py

[山水夜止]

zIT

Thanks for your prompt reply! I solve the problem by using the mirror website of hugging face. HF_ENDPOINT=https://hf-mirror.com python train_lora.py

Thanks for your prompt reply! I solve the problem by using the mirror website of hugging face. HF_ENDPOINT=https://hf-mirror.com python train_lora.py

Thank you. It works.