Video Prediction Visualization¶

We provide benchmark results of spatiotemporal prediction learning (STL) methods on popular traffic prediction datasets. More STL methods will be supported in the future. Issues and PRs are welcome! Visualization of GIF is released.

Table of Contents¶

Visualization of Moving MNIST Benchmarks
Visualization of Moving FashionMNIST Benchmarks
Visualization of Moving MNIST-CIFAR Benchmarks
Visualization of KittiCaltech Benchmarks
Visualization of KTH Benchmarks
Visualization of Human 3.6M Benchmarks

Currently supported spatiotemporal prediction methods

[x] ConvLSTM (NeurIPS’2015)
[x] PredNet (ICLR’2017)
[x] PredRNN (NeurIPS’2017)
[x] PredRNN++ (ICML’2018)
[x] E3D-LSTM (ICLR’2018)
[x] MIM (CVPR’2019)
[x] CrevNet (ICLR’2020)
[x] PhyDNet (CVPR’2020)
[x] MAU (NeurIPS’2021)
[x] PredRNN.V2 (TPAMI’2022)
[x] SimVP (CVPR’2022)
[x] SimVP.V2 (ArXiv’2022)
[x] TAU (CVPR’2023)
[x] DMVFN (CVPR’2023)

Currently supported MetaFormer models for SimVP

[x] ViT (ICLR’2021)
[x] Swin-Transformer (ICCV’2021)
[x] MLP-Mixer (NeurIPS’2021)
[x] ConvMixer (Openreview’2021)
[x] UniFormer (ICLR’2022)
[x] PoolFormer (CVPR’2022)
[x] ConvNeXt (CVPR’2022)
[x] VAN (ArXiv’2022)
[x] IncepU (SimVP.V1) (CVPR’2022)
[x] gSTA (SimVP.V2) (ArXiv’2022)
[x] HorNet (NeurIPS’2022)
[x] MogaNet (ArXiv’2022)

We provide visualization figures of various video prediction methods on various benchmarks. You can plot your own visualization with tested results (e.g., work_dirs/exp_name/saved) by vis_video.py. Note that --vis_dirs denotes visualize all experimental folders under the path, and --vis_channel can select the channel for visualization. For example, run plotting with the script:

python tools/visualizations/vis_video.py -d mmnist -w work_dirs/exp_name --index 0 --save_dirs fig_mmnist_vis

Visualization of Moving MNIST Benchmarks¶

We provide benchmark results on the popular Moving MNIST dataset using \(10\rightarrow 10\) frames prediction setting in configs/mmnist.

ConvLSTM

DMVFN

E3D-LSTM

MAU

MIM

PhyDNet

PredRNN

PredRNN++

PredRNN-V2

SimVP-V1

SimVP-V2

TAU

SimVP-ConvMixer

SimVP-ConvNeXt

SimVP-HorNet

SimVP-MLPMixer

SimVP-MogaNet

SimVP-Poolformer

SimVP-Swin

SimVP-Uniformer

SimVP-VAN

SimVP-ViT

(back to top)

Visualization of Moving FMNIST Benchmarks¶

Similar to Moving MNIST, we also provide the advanced version of MNIST, i.e., MFMNIST benchmark results, using \(10\rightarrow 10\) frames prediction setting in configs/mfmnist.

ConvLSTM

E3D-LSTM

MAU

MIM

PhyDNet

PredRNN

PredRNN++

PredRNN-V2

SimVP-V1

SimVP-V2

TAU

SimVP-ConvMixer

SimVP-ConvNeXt

SimVP-HorNet

SimVP-MLPMixer

SimVP-MogaNet

SimVP-Poolformer

SimVP-Swin

SimVP-Uniformer

SimVP-VAN

SimVP-ViT

(back to top)

Visualization of Moving MNIST-CIFAR Benchmarks¶

Similar to Moving MNIST, we further design the advanced version of MNIST with complex backgrounds from CIFAR-10, i.e., MMNIST-CIFAR benchmark, using \(10\rightarrow 10\) frames prediction setting in configs/mmnist_cifar.

ConvLSTM