nzw0301 / Understanding-Negative-Samples

#NeurIPS2021 Codes for Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Codes of Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning

Create experimental env

  • Linux machine with four GPUs
  • Conda
  • Python's dependencies: this file:
conda create --name research --file spec-file.txt
conda activate research

External dependency

Apex install

git clone git@github.com:NVIDIA/apex.git
cd apex
git checkout 4ef930c1c884fdca5f472ab2ce7cb9b505d26c1a  # to fix the exact library version
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./

Detailed version of PyTorch

conda install pytorch==1.6.0 torchvision==0.7.0 cudatoolkit=10.2 -c pytorch

Preparation

In NLP experiments, we need similar words constructed from fasttext's pre-trained word embeddings.

cd code
wget https://dl.fbaipublicfiles.com/fasttext/vectors-english/crawl-300d-2M.vec.zip
python nlp_construct_simiar_data.py

Please fill the value of replace_data in conf/dataset/ag_news.yaml with the generated file's path. For example,

replace_data: /home/nzw/code/ag_replace_ids.npy

Training

Please run content of all scripts in ./scripts/**/train/ under code.

Evaluation

After training, please run ./gather_weights.py to generate text files for evaluation.

Please run content of all scripts in ./scripts/**/eval/ under code as well.

For the AG news dataset, please run code/notebooks/filter_ag_news.ipynb after evaluation of mean classifier before the other evaluation scripts such as linear classifier and bound computation.

To obtain all figures and tables in the paper, you run notebooks in code/notebooks/. The codes save generated figures and tables into ./doc/figs and ./doc/tabs, respectively.

Related resources

Reference

@inproceedings{NS2021,
    title     = {Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning},
    author    = {Kento Nozawa, Issei Sato},
    year      = {2021},
    booktitle = {NeurIPS},
    pages     = {5784--5797},
}

About

#NeurIPS2021 Codes for Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning

License:MIT License


Languages

Language:Jupyter Notebook 83.1%Language:Python 14.5%Language:Shell 2.5%