FarmingTong / ChronoMagic-Bench

ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generation

ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of

Text-to-Time-lapse Video Generation

If you like our project, please give us a star ⭐ on GitHub for the latest update.

This repository is the official implementation of ChronoMagic-Bench, a benchmark for metamorphic evaluation of text-to-time-lapse video generation. The key insight is to evaluate the capabilities of Text-to-Video Generation Models in physics, biology, and chemistry by enabling the generation of time-lapse videos, which are characterized by rich physics priors, through a free-form text prompt.

💡 We also have other video generation project that may interest you ✨.

Open-Sora-Plan
PKU-Yuan Lab and Tuzhan AI etc.

MagicTime
Shenghai Yuan, Jinfa Huang and Yujun Shi etc.

📣 News

• ⏳⏳⏳ Evaluate more Text-to-Video Generation Models via ChronoMagic-Bench.
• ⏳⏳⏳ Release the code of the "Multi-Aspect Data Preprocessing", which is used to process the dataset. The code is being organized.
• ⏳⏳⏳ Support evaluating customized videos. The code and instructions are being organized.
• [2024.06.28] 🔥 We released the ChronoMagic-Pro and ChronoMagic-ProH datasets. The datasets include 460K and 150K time-lapse video-text pairs respectively and can be downloaded at HF-Dataset-Pro and HF-Dataset-ProH.
• [2024.06.27] 🔥 We release the arXiv paper and Leaderboard for ChronoMagic-Bench, and you can click here to read the paper and here to see the leaderboard.
• [2024.06.26] 🔥 We release the testing prompts, reference videos and generated results by different models in ChronoMagic-Bench, and you can click here to see more details.
• [2024.06.25] 🔥 All codes & datasets are coming soon! Stay tuned 👀!

😮 Highlights

ChronoMagic-Bench can reflect the physical prior capacity of Text-to-Video Generation Model.

📣 Overview

In contrast to existing benchmarks, ChronoMagic-Bench emphasizes generating videos with high persistence and strong variation, i.e., metamorphic time-lapse videos with high physical prior content.

Backbone	Type	Visual Quality	Text Relevance	Metamorphic Amplitude	Temporal Coherence
UCF-101	General	✔️	✔️	❌	❌
Make-a-Video-Eval	General	✔️	✔️	❌	❌
MSR-VTT	General	✔️	✔️	❌	❌
FETV	General	✔️	✔️	❌	✔️
VBench	General	✔️	✔️	❌	✔️
T2VScore	General	✔️	✔️	❌	❌
ChronoMagic-Bench	Time-lapse	✔️	✔️	✔️	✔️

We specifically design four major categories for time-lapse videos (as shown below), including biological, human-created, meteorological, and physical videos, and extend these to 75 subcategories. Based on this, we construct ChronoMagic-Bench, comprising 1,649 prompts and their corresponding reference time-lapse videos.

Biological	Human Created	Meteorological	Physical
"Time-lapse of microgreens germinating and growing ..."	"Time-lapse of a modern house being constructed in ..."	"Time-lapse of a beach sunset capturing the sun's ..."	"Time-lapse of an ice cube melting on a solid ..."
"Time-lapse of microgreens germinating and growing ..."	"Time-lapse of a 3D printing process: starting with ..."	"Time-lapse of a solar eclipse showing the moon's ..."	"Time-lapse of a cake baking in an oven, depicting ..."
"Time-lapse of a butterfly metamorphosis from ..."	"Time-lapse of a busy nighttime city intersection ..."	"Time-lapse of a landscape transitioning from a ..."	"Time-lapse of a strawberry rotting: starting with ..."

🎓 Evaluation Results

We visualize the evaluation results of various open-source and closed-source T2V generation models across ChronoMagic-Bench.

🏆 Leaderboard

See numeric values at our Leaderboard 🥇🥈🥉

⚙️ Requirements and Installation

We recommend the requirements as follows.

Environment

git clone https://github.com/PKU-YuanGroup/ChronoMagic-Bench.git
cd ChronoMagic-Bench
conda create -n chronomagic python=3.10
conda activate chronomagic

# install base packages
pip install -r requirements.txt

# install flash-attn
git clone https://github.com/Dao-AILab/flash-attention.git
cd flash-attention/csrc/layer_norm && pip install .
cd ../../../
rm -r flash-attention

👍 Acknowledgement

• This project wouldn't be possible without the following open-sourced repositories: CoTracker, InternVideo2, UMT, FETV, VBench, ShareGPT4Video and LAION Aesthetic Predictor.

🔒 License

• The majority of this project is released under the Apache 2.0 license as found in the LICENSE file.
• The service is a research preview. Please contact us if you find any potential violations.

✏️ Citation

If you find our paper and code useful in your research, please consider giving a star ⭐ and citation 📝.

@misc{yuan2024chronomagicbenchbenchmarkmetamorphicevaluation,
      title={ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generation}, 
      author={Shenghai Yuan and Jinfa Huang and Yongqi Xu and Yaoyang Liu and Shaofeng Zhang and Yujun Shi and Ruijie Zhu and Xinhua Cheng and Jiebo Luo and Li Yuan},
      year={2024},
      eprint={2406.18522},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2406.18522}, 
}

🤝 Contributors