Kyfafyd / MirrorGAN

Reproduction of the paper MirrorGAN: Learning Text-to-image Generation by Redescription

This repo is under construction

---

This repo is an unofficial reproduction of paper MirrorGAN: Learning Text-to-image Generation by Redescription

Getting Started

Dependencies

• Python == 2.7.12
• torch == 0.4.1
• torchvision == 0.2.0

Dataset

Official Dataset

• birds

Your Own Dataset

• Dataset Directory

  |- dataset
  |   |---Your Dataset Name
  |   |   |---images
  |   |   |   |---category1
  |   |   |   |   |---aaa.jpg
  |   |   |   |   |---bbb.jpg
  |   |   |   |   |---...
  |   |   |   |---category2
  |   |   |   |   |---aaa.jpg
  |   |   |   |   |---bbb.jpg
  |   |   |   |   |---...
  |   |   |   |---...
  |   |   |---resized
  |   |   |   |---category1
  |   |   |   |   |---aaa.jpg
  |   |   |   |   |---bbb.jpg
  |   |   |   |   |---...
  |   |   |   |---category2
  |   |   |   |   |---aaa.jpg
  |   |   |   |   |---bbb.jpg
  |   |   |   |   |---...
  |   |   |   |---...
  |   |---text
  |   |   |---category1
  |   |   |   |---aaa.text
  |   |   |   |---bbb.text
  |   |   |   |---...
  |   |   |---category2
  |   |   |   |---aaa.text
  |   |   |   |---bbb.text
  |   |   |   |---...
  |   |   |---...
  |   |---train
  |   |   |---filenames.pickle
  |   |---test
  |   |   |---filenames.pickle
  |   |---captions.json
  

• Preprocess Dataset

  Generate resized/, text/, train/, test/, captions.json

  python2 preprocess.py

Pretrained STEM/STREAM

Below pretrained models are for birds dataset, if you want to apply MirrorGAN to your own dataset, please follow preprocess.py and pretrain section.

STEM

• image_encoder.pth
• text_encoder.pth

STREAM

• decoder.ckpt
• encoder.ckpt

Train/Test

Pretrain

# STEM
cd GLAM
python2 pretrain_DAMSM.py

# STREAM
cd STREAM
python2 resize.py
python2 build_vocab.py
python2 train.py

Train

cd GLAM
python2 mian.py

Test

cd GLAM
python2 mian.py --cfg cfg/eval_bird.yml

Contact

If you have any questions, please feel free to contact ZHAO WANG

Reference

MirrorGAN: Learning Text-to-image Generation by Redescription

AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks

Image Captioning