zzwei1 / SimVPv2

The official implementation of the paper SimVP: Towards Simple yet Powerful Spatiotemporal Predictive learning.

SimVP: Towards Simple yet Powerful Spatiotemporal Predictive learning

📘Documentation | 🛠️Installation | 🚀Model Zoo | 🆕News

This repository is an open-source project for video prediction benchmarks, which contains the implementation code for paper:

SimVP: Towards Simple yet Powerful Spatiotemporal Predictive learning
Cheng Tan, Zhangyang Gao, Siyuan Li, Stan Z. Li.

Introduction

This is the journal version of our previous conference work (SimVP: Simpler yet Better Video Prediction, In CVPR 2022).

It is worth noticing that the hidden Translator $h$ in SimVP can be replaced by any MetaFormer block (satisfying the macro design of token mixing and channel mixing).

The performance of SimVPs on the Moving MNIST dataset. For the training time, the less the better. For the inference efficiency (frames per second), the more the better.

Quantitative results of different methods on the Moving MNIST dataset ($10 \rightarrow 10$ frames).

(back to top)

Overview

• simvp/api contains an experiment runner.
• simvp/core contains core training plugins and metrics.
• simvp/datasets contains datasets and dataloaders.
• simvp/methods/ contains training methods for various video prediction methods.
• simvp/models/ contains the main network architectures of various video prediction methods.
• simvp/modules/ contains network modules and layers.
• tools/non_dist_train.py is the executable python file with possible arguments for training, validating, and testing pipelines.

News and Updates

[2023-02-18] SimVP v0.1.0 is released. Benchmark results and config files are updated for MMNIST, TaxiBJ, and WeatherBench datasets.

Installation

This project has provided an environment setting file of conda, users can easily reproduce the environment by the following commands:

git clone https://github.com/chengtan9907/SimVPv2
cd SimVPv2
conda env create -f environment.yml
conda activate SimVP
python setup.py develop

Dependencies

• argparse
• fvcore
• numpy
• hickle
• scikit-image=0.16.2
• scikit-learn
• torch
• timm
• tqdm

Please refer to install.md for more detailed instructions.

Getting Started

Please see get_started.md for the basic usage. Here is an example of single GPU training SimVP+gSTA on Moving MNIST dataset.

bash tools/prepare_data/download_mmnist.sh
python tools/non_dist_train.py -d mmnist --lr 1e-3 -c ./configs/mmnist/simvp/SimVP_gSTA.py --ex_name mmnist_simvp_gsta

(back to top)

Overview of Model Zoo

We support various video prediction methods and will provide benchmarks on various video prediction datasets. We are working on add new methods and collecting experiment results.

• Video Prediction Methods.

  Currently supported methods

  • ConvLSTM (NIPS'2015)
  • PredRNN (NIPS'2017)
  • PredRNN++ (ICML'2018)
  • E3D-LSTM (ICLR'2018)
  • MAU (CVPR'2019)
  • CrevNet (ICLR'2020)
  • PhyDNet (CVPR'2020)
  • PredRNN.V2 (TPAMI'2022)
  • SimVP (CVPR'2022)
  • SimVP.V2 (ArXiv'2022)
  Currently supported MetaFormer models for SimVP

  • ViT (ICLR'2021)
  • Swin-Transformer (ICCV'2021)
  • MLP-Mixer (NIPS'2021)
  • ConvMixer (Openreview'2021)
  • UniFormer (ICLR'2022)
  • PoolFormer (CVPR'2022)
  • ConvNeXt (CVPR'2022)
  • VAN (ArXiv'2022)
  • IncepU (SimVP.V1) (CVPR'2022)
  • gSTA (SimVP.V2) (ArXiv'2022)
  • HorNet (NIPS'2022)
  • MogaNet (ArXiv'2022)

• Video Prediction Benchmarks.

  Currently supported datasets

  • KTH Action (ICPR'2004) [download] [config]
  • KittiCaltech Pedestrian (IJRR'2013) [download] [config]
  • Moving MNIST (ICML'2015) [download] [config]
  • TaxiBJ (AAAI'2017) [download] [config]
  • WeatherBench (ArXiv'2020) [download] [config]

(back to top)

License

This project is released under the Apache 2.0 license. See LICENSE for more information.

Acknowledgement

SimVPv2 is an open-source project for video prediction methods created by researchers in CAIRI AI Lab. We encourage researchers interested in video and weather prediction to contribute to SimVPv2! We borrow the official implementations of ConvLSTM, PredRNN variants, E3D-LSTM, MAU, CrevNet, and PhyDNet.

Citation

If you are interested in our repository and our paper, please cite the following paper:

@article{tan2022simvp,
  title={SimVP: Towards Simple yet Powerful Spatiotemporal Predictive Learning},
  author={Tan, Cheng and Gao, Zhangyang and Li, Stan Z},
  journal={arXiv preprint arXiv:2211.12509},
  year={2022}
}

Contribution and Contact

For adding new features, looking for helps, or reporting bugs associated with SimVPv2, please open a GitHub issue and pull request with the tag "help wanted" or "enhancement". Feel free to contact us through email if you have any questions. Enjoy!

• Cheng Tan ([e-mail]), Westlake University & Zhejiang University
• Siyuan Li ([e-mail]), Westlake University & Zhejiang University
• Zhangyang Gao ([e-mail]), Westlake University & Zhejiang University

(back to top)