TemporalKit

An all in one solution for adding Temporal Stability to a Stable Diffusion Render via an automatic1111 extension

You must install FFMPEG to path before running this

You can find a demonstration run through with a single batch here:
https://twitter.com/CiaraRowles1/status/1645923461343363072

And a batch demonstration here:
https://mobile.twitter.com/CiaraRowles1/status/1646458056803250178

Ebsynth tutorial:
https://twitter.com/CiaraRowles1/status/1648462374125576192
NOTE: EBSYNTH DOES NOT REGISTER THE KEYFRAMES IF YOU USE ABOVE 20.

Ebsynth split frames tutorial:
https://www.youtube.com/watch?v=z3YNHiuvxyg&ab_channel=CiaraRowles

Example results you can get:

consistenanimation.mp4

The values in the extension are as follows

Variable	Description
`FPS`	The fps the video is extracted and produced at.
`batch_Size`	This is the number of frames between each keyframe, so for example if you had an fps of 30, and a batch size of 10, it would make 3 keyframes a second and estimate the rest.
`per side`	This is the square root of the number of frames per plate, so for example a per side value of 2 would make 4 plates, 3, 9 plates, 4 16 plates.
`Resolution`	The size of each plate, it is strongly reccomended you set this to a multiple of your per side variable
`batch settings`	Only open this drop down if you want to generate a folder of plates.
`Max Frames`	When generating a folder of plates, this gets how many frames at the above fps you want to get, and then divides them into plates in groups of (per side * per side * batch size)
`Border Frames`	Every batch generated plate will contain this many frames from the next plate and blend between them.
`Batch Folder`	If you're generating a batch of plates, just specify a empty folder and on clicking run, it will populate it with the relevant folders and files, all you need to do is go to img2img batch processing in original sd, enter the newly create input folder as the input, the newly created output folder as the output, generate, move back to the temporal-kit Batch-Warp Tab, put in the whole folder directory and click read and it will set everything up.
`Output Resolution`	The maximum resolution on any side of the output video.

FAQ:

Q: My video has smearing.

A: Use a higher fps and/or lower batchnumber, the closer together the keyframes the less artifacts.

Q: Stable diffusion cannot be turned on after installing this Extension. ModuleNotFoundError: No module named 'tqdm.auto'

A: Because the dependency currently used by this plug-in uses an old version of tqdm, an error occurs. The short-term solution is to manually install a new version(4.66.1) of tqdm.
pip install tqdm==4.66.1
More information

Step-by-Step Tutorial

Written with reference to this web page teaching and my own(cocomine) experience

Step 1: Install Extensions on WebUI

Open Extensions tab > Install from URL > Paste follow link in URL for extension’s git repository > Click Install

https://github.com/CiaraStrawberry/TemporalKit

Step 2: Install FFMPEG

Ubuntu

sudo apt install ffmpeg

Arch Linux

sudo pacman -S ffmpeg

Windows

Download and install from https://ffmpeg.org/download.html Make sure to add ffmpeg to your PATH. Learn more: https://www.wikihow.com/Install-FFmpeg-on-Windows

Step 4: Prepare your video

Create a folder in your desired location. This folder will be used to store the files that need to be processed. Prepare the video you need to use (referred to as the original video in subsequent teaching), and understand the format of the original video, such as resolution and frame.

Step 5: Extract frames from the original video

Open Temporal-Kit Tab on Top.
Open Pre-Process Tab.
Drag & Drop the original video into the Input Video.
Set fps to the frame rate of the original video.
Set frames per keyframe to the number of frames between each keyframe. For example, if the original video is 30fps and you set it to 10, then 3 keyframes will be generated per second, and the rest will be estimated.
Set Side to the square root of the number of frames per plate. For example, if you set it to 2, 4 plates(2x2) will be generated, 3, 9 plates(3x3), 4, 16 plates(4x4).
Set Height Resolution to the size of each plate. It is strongly recommended that you set this to a multiple of your side variable.
For example, if you want to generate 4 plates and set the side to 2, each plate high 512, then you need to set the height resolution to 1024(512x2).
Set Target Folder to the folder you created in step 4.
Open Batch Settings Tab.
Tick the Batch Run.
Open EBsynth Tab.
Tick the Split Video

When you complete the above steps you should see a structure similar to this in the folder you specified (depending on the length of your video)

You will see a structure like this in the video clips divided into folders named by numbers.

If you encounter out of memory issue in the next img2img step, reduce the side or Height Resolution parameters.

Step 6: Perform Img2img on keyframes

Let’s deal with the folder named 0 first.
Go to the Img2img page. Switch to the Batch tab. Set the following parameters:

Input directory: The name of your target directory with \input appended. E.g. YOUR_FOLDER_PATH_IN_SETP_4\0\input
Output directory: Similarly but with \output appended. E.g. YOUR_FOLDER_PATH_IN_SETP_4\0\output

Enter a prompt and a negative prompt like txt2img.
Sampling method: DPM++2M Karras
Sampling steps: 20
CFG scale: 7
Denoising strength: 0.5 (adjust accordingly)

The above parameters can be changed as needed.

Control Net (option, which would give better results if available)

In ControlNet (Unit 0) section, set:

Enable: Yes
Pixel Perfect: Yes
ControlType: Tile
Preprocessor: tile_resample
Model: control_xxxx_tile

Press Generate. After it is done, you will find the image in the batch output folder.

Make sure to open the image in full size and inspect the details in full size. Make sure they look sharp and have a consistent style.

If you want to obtain high-resolution images, please put the output images back into img2img, and adjust resizd by 1.5-2.0, Denoising strength 0.3-0.4, and then generate.

Step 7: Prepare EbSynth data

Go to Temporal-Kit page and switch to the Ebsynth-Process tab.

Input Folder: Put in the same target folder path you put in the Pre-Processing page. E.g. YOUR_FOLDER_PATH_IN_SETP_4\0

Click read last_settings. If your input folder is correct, the video and the settings will be populated.

Click prepare ebsynth. After it is done, you should see the keys folder populated with your stylized keyframes, and the frames folder populated with your images.

Please note that this program does not generate .ebs files. When your images are imported into the program, they will be automatically populated.

Step 8: Process with EbSynth

Now open the EbSynth program.

Open the File Explorer and navigate to the folder your creation in step4. You should folder like the ones showed below. We need the keys folder and the frames folder for EbSynth.

Drag the keys folder from the File Explorer and drop it to the Keyframes field in EbSynth.
Drag the frames folder from the File Explorer and drop it to the frames field in EbSynth.

Click Run All and wait for them to complete. When it is done, you should see a series of out_##### directories generated in the target project folder.

Please download the program from the official website.
https://ebsynth.com/

Step 9: Generate the final video

Now go back to AUTOMATIC1111(webUI). You should still be on the Temporal Kit page and Ebsynth-Process tab.

Click recombine ebsynth and you are done!

Look how smooth the video is. With some tweaking, you can probably make it better!

Step 10: Continue processing other split segments

Repeat steps 6 to 9. Process other folders (1,2,3,4,...)

Step 11: Combine the split videos

Use a video splicing program to merge segmented videos

TODO

set up diffusion based upscaling for the plates output
get the img2img button working with batch processing.
add a check to see if the output folder was added.
fix that weird shutdown error it gives after running
hook up to the api.
flowmaps from game engine export\import support

Thanks to RAFT for the optical flow system.

martjay / TemporalKit