Giters

facebookresearch / MetaCLIP

ICLR2024 Spotlight: curation/training code, metadata, distribution and pre-trained models for MetaCLIP; CVPR 2024: MoDE: CLIP Data Experts via Clustering

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Which OpenClip should be used

shkarupa-alex opened this issue · comments

commented

It is not clear from readme:

  1. Did you use your copy of OpenClip (from src) or OpenClip from pypi?
  2. Why did you decided to use QuickGelu if it is slower and much memory-hungry?
commented

1, good question, we use our copy of OpenCLIP and customize the setup to OpenAI CLIP (that was never done before).
2, we aim for controlled experiments over OpenAI CLIP so all gains are from data ONLY for fair comparison on data and all data algorithms in future.
All existing non OpenAI works are not changing data alone: big batch size, model, lr. These are CLIP "system" that can always combine w/ latest NN tech or hardware; these are not very related to CLIP itself (eg changing activation function are common to every transformer).
BTW, we notice slightly better acc. perf. on quickgelu over gelu in 2nd half of training (not initially); so we suspect the benefits of gelu is not fully verified but very ad-hoc?