Implementation of a zero shot classification model using the cifar-10 dataset.
Simple zero shot classification algorithm that allows a model to make predictions on images that aren't within the class domains of the training dataset. The model performs a matrix multiplication using semantic space vectors (glove word embeddings) of training class names in the final layer, where the final output is of size (batch_dim, class_count). In this sense, the training of the model works in the same way as an image classifier, where a NLL loss is used with correct image labels in order to train model weights. At test time, the outputs of the second last layer are extracted from the model, and used in nearest neighbour search with semantic vectors of seen and unseen classes for classification (l2 distance).
Ultimately, the model aims to learn an efficient mapping of an image space to a semantic space (e.g. the picture shown above).
Zero Shot Classes Used:
- Bicycle, Helicopter, Submarine
Achieves 90+% accuracy for top 3 predictions with zsl classes (true labels were within the top 3 predictions of the model - true label was amongst the closest 3 vectors to the feature prediction), and 40%+ for top 1 prediction.
CNN depth restricted due to memory constraints on GPU.
Uses glove vectors from https://nlp.stanford.edu/projects/glove/ ZSL classes taken from google via scraper.
Packages Required in Environment:
- Tensorflow
- Numpy
- Matplotlib
- CV2
GPU training is preferred.
To execute the program, use the following command whilst in terminal:
python main.py