β totally no tuning β less than 20% extra time β support 512 frames
Haonan Qiu, Menghan Xia*, Yong Zhang, Yingqing He,
Xintao Wang, Ying Shan, and Ziwei Liu*
(* corresponding author)
From Tencent AI Lab and Nanyang Technological University.
![](https://raw.githubusercontent.com/acampi24/LongerCrafter/main/assets/t2v/hd01.gif)
Input: "A chihuahua in astronaut suit floating in space, cinematic lighting, glow effect";
Resolution: 1024 x 576; Frames: 64.
![](https://raw.githubusercontent.com/acampi24/LongerCrafter/main/assets/t2v/hd02.gif)
Input: "Campfire at night in a snowy forest with starry sky in the background";
Resolution: 1024 x 576; Frames: 64.
π€π€π€ LongerCrafter (FreeNoise) is a tuning-free and time-efficient paradigm for longer video generation based on pretrained video diffusion models.
- [2023.10.24]: π₯π₯ Release the LongerCrafter (FreeNoise), longer video generation!
- [2023.10.25]: π₯π₯ Release the 256x256 model and support multi-prompt generation!
Model | Resolution | Checkpoint | Description |
---|---|---|---|
VideoCrafter (Text2Video) | 320x512 | Temporarily Unavailable | Support 128 frames on NVIDIA A100 (40GB) |
VideoCrafter (Text2Video) | 576x1024 | Hugging Face | Support 64 frames on NVIDIA A100 (40GB) |
VideoCrafter (Text2Video) | 256x256 | Hugging Face | Support 512 frames on NVIDIA A100 (40GB) |
(Reduce the number of frames when you have smaller GPUs, e.g. 256x256 resolutions with 64 frames.)
conda create -n freenoise python=3.8.5
conda activate freenoise
pip install -r requirements.txt
- Download pretrained T2V models via Hugging Face, and put the
model.ckpt
incheckpoints/base_1024_v1/model.ckpt
. - Input the following commands in terminal.
sh scripts/run_text2video_freenoise_1024.sh
- Download pretrained T2V models via Hugging Face, and put the
model.pth
incheckpoints/base_256_v1/model.pth
. - Input the following commands in terminal.
sh scripts/run_text2video_freenoise_mp_256.sh
FreeNoise is supposed to work on other similar frameworks. An easy way to test compatibility is by shuffling the noise to see whether a new similar video can be generated (set eta to 0). If your have any questions about applying FreeNoise to other frameworks, feel free to contact Haonan Qiu.
VideoCrafter: Framework for high-quality video generation.
ScaleCrafter: Tuning-free method for high-resolution image/video generation.
TaleCrafter: An interactive story visualization tool that supports multiple characters.
@misc{qiu2023freenoise,
title={FreeNoise: Tuning-Free Longer Video Diffusion Via Noise Rescheduling},
author={Haonan Qiu and Menghan Xia and Yong Zhang and Yingqing He and Xintao Wang and Ying Shan and Ziwei Liu},
year={2023},
eprint={2310.15169},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
We develop this repository for RESEARCH purposes, so it can only be used for personal/research/non-commercial purposes.