Finetune the released MetaCLIP models on my own dataset
ShuxunoO opened this issue · comments
ShuxunoO commented
Hello~
How can I finetune the released MetaCLIP models on my own dataset(Img-Txt paires) and how do I organize the data?
Hu Xu commented
Thx, for your interests in this project. It can be very similar to pre-training and here's a suggested steps:
- check the pre-training setup: https://github.com/facebookresearch/MetaCLIP#training
- derive a new config from
b32_400m
and add `pretrained="metaclip_400m"
def b32_400m_ft():
return b32_400m(
pretrained="metaclip_400m", # should download pre-trained MetaCLIP ViT-B/32 400M
)
(we will have a new config system released soon)
- depends on your fine-tuning set format, properly set
train_data
forhttps://github.com/facebookresearch/MetaCLIP/blob/main/src/training/data.py
and we highly recommend use your own dataloader to control randomness instead of built-in one.