R_Unet
This project applies recurrent method upon U-net to perform pixel level video frame prediction.
Part of our result is published at IEEE GCCE 2020. pdf.
Brief introduction
Taking advantage of LSTM and U-net encode-decoder, we wish to be able in predicting next (n) frame(s).
Currently using a 2 layer LSTM network (V1) or convolution LSTM (V2) as RNN network applying on latent feature of U net
In our latest v4 model, we use convolutional LSTM in each level and take short cut in v2 out
On the other hand, we are now using v4_mask model to train mask, image input and mask, image prediction output
This model holds same structure as v4 but simply change output layer to output mask tensor.
Usage
- configuration: config.json
- parse configuration: class parse_arguement.py
- training file: train.py
- V1 model: R_Unet_v1.py
- V2 model: R_Unet_ver_2.py
- V4 model: R_Unet_ver_4.py
to train v1 model: python3 train.py config
to train other model: python3 train_v2.py config
Our Model Architecture
Current we are working on a better model using convolution lstm, name as runet_v2
-
model v1:
-
model v2:
-
model v4:
Some result
prediction:
Ground truth:
mask prediction
prediction:
Ground truth:
References
[1] Stochastic Adversarial Video Prediction, CVPR 2018
[2] High Fidelity Video Prediction with
Large Stochastic Recurrent Neural Networks, NeurIPS 2019
[3] convLSTM - The convolution lstm framework used
Hsu Mu Chien, Watanabe Lab, Department of Fundamental Science and Engineering, Waseda University.