This repo is implementations of neural recipe generators using PyTorch.
Now we implemented the following 5 models:
- Images2seq (https://www.aclweb.org/anthology/N16-1147v2.pdf)
- GLAC Net (https://arxiv.org/pdf/1805.10973.pdf)
- Retrieval Attention (RetAttn) (https://www.aclweb.org/anthology/W19-8650.pdf)
- SSiD (https://www.aclweb.org/anthology/P19-1606.pdf)
- SSiL (https://www.aclweb.org/anthology/P19-1606.pdf)
Note We could not implement the SSiD and SSiL perfectly due to lack of details of a finite state machine (FSM).
- Python 3.7
- CUDA 10.2 and cuDNN v7.6
- PyTorch 1.5.0
- install other required modules
pip install -r requirements.txt
The dataset used in this repo is the story boarding dataset (https://www.aclweb.org/anthology/P19-1606.pdf). As mentioned here, the original scripts did not save the train/val/test splits. Thus, this scripts lead you to download the data from instructable.com and snapguide.com and split them into train/val/test datasets.
Follow this repo.
Then, Copy instructables.json
and snapguide.json
to data/
directory.
The following scripts lead you to split the dataset with train/val/test.
./make_directory.sh
cd preprocess
python build_dataset.py -d ./data/ --dl
python convert_pickles_into_trainable_format.py -d ./data -o ./data/features/
Exsisting 5 recipe generation models are divided into two types: scratch models (Images2seq, GLAC Net) and pretraining-based models (SSiD, SSiL, RetAttn).
Note Training scripts are under construction. Now I only implemented the GLAC Net.
Scratch models learn to generate a recipe from random weights. You can train these models as:
python train.py -m {glacnet/images2seq}
[WIP]
[WIP]
@misc{taichi19recipe,
author = {Taichi Nishimura},
title = {recipe_generation_from_an_image_sequence.pytorch},
howpublished = {https://github.com/misogil0116/recipe_generation_from_an_image_sequence.pytorch},
year = {2020}
}