facebookresearch / MetaCLIP

ICLR2024 Spotlight: curation/training code, metadata, distribution and pre-trained models for MetaCLIP; CVPR 2024: MoDE: CLIP Data Experts via Clustering

How to implement English Language IDentification (LID)？

hbchen121 opened this issue 4 months ago · comments

Cheng Nan commented 4 months ago

How do you implement the function o "English Language IDentification (LID)" in the paper?

Thanks~ Looking forward to your reply

Hu Xu commented 4 months ago

In the paper we use a production LID model. In this open-source version you can try FastText:

MetaCLIP/metaclip/cc_matching.py

Line 24 in 43ba9bc

def lid(text):