Pretrained text encoder

[ethan cohen]

Is it possible to use and train dalle with an external ( frozen) text encoder ( as those available in hugging face) ?

[ethan cohen]

Anyone has an idea about this ? @lucidrains

[Ronghuan Wu]

Hi. If you want to use pretrained language model, you are actually using the text embedding of that model.

1. At first, you can load and save the text embedding layer weight of pretrained models like CLIP and BERT.
2. Then, you need to replace the text_emb in DALLE __init__ function. Now, instead of using nn.Embedding to create new text embedding, you can use torch.load to load pretrained weight saved in step 1.

Example: link