GeNIe: Generative Hard Negative Images Through Diffusion

This Repository is an official implementation of GeNIe. Our code for generating samples is based on BOOMERANG and for few-shot testing is based on UNISIAM.

Overview

GeNIe is a novel data augmentation technique employing Generative text-based latent Diffusion models. GeNIe merges contrasting data points (an image from the source category and a text prompt from the target category) using a latent diffusion model conditioned on the text prompt. By limiting diffusion iterations, we preserve low-level and background features from the source image while representing the target category, creating challenging samples. Additionally, our method further enhances its effectiveness by dynamically adjusting the noise level for each image, known as GeNIe-Ada. Rigorous experimentation across both few-shot and long-tail distribution scenarios substantiates the superior performance of our approach over existing techniques.

Visualizing Generated Samples

GeNIe effectively generates hard negatives for the source image class by preserving low-level features and transforming its main target class according to the prompt.

Noise Adaptive Sampling

A small noise ratio generates an image that closely resembles the source. Conversely, when the noise ratio is large, it resembles the target category's semantics. Thus, to choose the optimal ratio adaptively for each source image to generate images that preserve the low-level semantics of the source image while effectively representing the semantics of the target category, we propose GeNIE-Ada. Genie-Ada traces the semantic trajectory from source image embeddings to target class embedding through the lens of the classifier f(.), to then adaptively select the sample right after the largest semantic shift.

Requirements

All our experiments use the PyTorch library. Install PyTorch and ImageNet datasets following the official PyTorch ImageNet training code. We used Python 3.7 for our experiments.

Getting Started

Please install dependencies in a virtual environment:

pip install transformers
pip install diffusers==0.19.0

Demo

You can try GeNIe with this colab GeNIe Colab.

Few-Shot on tiered-ImageNet and mini-ImageNet

Use the imagenet_sampler.ipynb notebook to generate episodes for mini-ImageNet and tiered-ImageNet
To generate augmentations for all baselines (txt2img, img2img, and GeNIe), utilize the generate_data.sh script as follows:

generate_data.sh 0 20 30

This script employs GPU 0 to generate augmentations for episodes 20 to 30.

To generate Noise Adaptive dataset for few-shot learning - run /few_shot/noise_adaptive.ipynb
Train on the augmented dataset:

CUDA_VISIBLE_DEVICES=0 python ./train.py --data_path /home/datadrive/mini_imagenet_fs --backbone resnet18 --eval_path /home/datadrive/mini_imagenet_fs/models/mini_r18_v2.pth --transform weak --caching_epochs 5 --n_shots 1 --clf LR --augs_name train_negative_noise_noad_r18_v3 &> PATH_TO_LOG.txt &

Change all paths and GPU/CUDA device IDs as per your settings.

Citation

If you make use of the code, please cite the following work:

@misc{koohpayegani2023genie,
      title={GeNIe: Generative Hard Negative Images Through Diffusion}, 
      author={Soroush Abbasi Koohpayegani and Anuj Singh and K L Navaneet and Hadi Jamali-Rad and Hamed Pirsiavash},
      year={2023},
      eprint={2312.02548},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

License

This project is under the MIT license.

UCDvision / GeNIe