Allen0307 / KKLIP-Knowledge-Distillation-Exploiting-K-means-Clustering-for-Language-Image-Pre-Training

Final project for the class 269 at 23 Fall

KKLIP: Knowledge Distillation Exploiting K-means Clustering for Language-Image Pre-Training

paper link

Authors: Chin-Lun Fu, Chun-Yao Chang, Kuei-Chun Kao, Nanyun (Violet) Peng

Overview

In this study, we introduces KKLIP, a novel approach designed to enhance the quality of CLIP by incorporating a new knowledge distillation (KD) method derived from Llama 2. Our method comprises three objectives: Text Embedding Distillation, Concept Learning, and Contrastive Learning.

Dataset

We use CC15M as our dataset. You can download all datasets from the website.

Pre-train

cd pre-train
python train-klip.py

Text Encoder Evaluation

python text_eval.py