awkrail / recipe_generation_from_an_image_sequence.pytorch

Recipe generation models from an image sequence (Pytorch implementation).

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool


This repo is implementations of neural recipe generators using PyTorch.
Now we implemented the following 5 models:

Note We could not implement the SSiD and SSiL perfectly due to lack of details of a finite state machine (FSM).


  1. Python 3.7
  2. CUDA 10.2 and cuDNN v7.6
  3. PyTorch 1.5.0
  4. install other required modules
pip install -r requirements.txt

Data Preparation

The dataset used in this repo is the story boarding dataset ( As mentioned here, the original scripts did not save the train/val/test splits. Thus, this scripts lead you to download the data from and and split them into train/val/test datasets.

1. Downloading the story boarding dataset

Follow this repo.
Then, Copy instructables.json and snapguide.json to data/ directory.

2. Preprocessing the dataset

The following scripts lead you to split the dataset with train/val/test.

cd preprocess
python -d ./data/ --dl
python -d ./data -o ./data/features/

Training and Validation

Exsisting 5 recipe generation models are divided into two types: scratch models (Images2seq, GLAC Net) and pretraining-based models (SSiD, SSiL, RetAttn).
Note Training scripts are under construction. Now I only implemented the GLAC Net.

Scratch models

Scratch models learn to generate a recipe from random weights. You can train these models as:

python -m {glacnet/images2seq}

Pretraining-based models





    author = {Taichi Nishimura},
    title = {recipe_generation_from_an_image_sequence.pytorch},
    howpublished = {},
    year = {2020}


Recipe generation models from an image sequence (Pytorch implementation).


Language:Python 99.5%Language:Shell 0.5%