Giters

backprop-ai / backprop

Backprop makes it simple to use, finetune, and deploy state-of-the-art ML models.

Home Page:https://backprop.co

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Finetuning Image Text Vectorizer with CLIP

singularity014 opened this issue · comments

commented

Hello, I tried finetuning Image-Text Vectorizer CLIP model using above approach. But I get stuck with the error -

image

Link to full code - Colab

What I need is something which gives cosine similarity between an image and a text, shall I finetune with triplet, or with cosine similarity? if its cosine similarity, then how will I get those cosine similarity?

The triplet variant takes text and image and gives one normalised vector, I am bit confused because I thought it would give a cosine similarity.

commented

Hey,

Your code is not public so I can't see it.
You should be able to finetune with either. Cosine similarity is just a function between two vectors.

See an example of it being used here: https://github.com/backprop-ai/backprop/blob/main/examples/ImageVectorisation.ipynb