This is Kenji's final year project with diffusion models. Research idea: If a model learns how to do something, it should be able to adapt its skills to different conditions.
The project's goal is to get a model to learn a motion and then see how the motion adapts to different conditions we set to the model (ie interacting with objects, moving along trajectories, changes in environment etc).
In this repo, there is
- Code for different architectures to turn a mocap motion as a diffusion model sampling process
- Vanilla DDPM with 1D diffusion (in the main branch)
- MDM (in the mdm branch)
- Diffuser (in the main branch)
- A way to apply constraints to the sampling process to alter the generated motion. A collection of notebooks showcasing different applications of motion diffusion models have been preparred
0_temporal_unet_diffusion.ipynb
- The base motion generation code1_motion_editing.ipynb
- A notebook experimenting with editing the motion by changing the joint positions2_starting_with_motion.ipynb
- A notebook experimenting with starting the sampling process with a motion instead of random noise3_long_projection.ipynb
- A notebook experimenting with sampling with containing more frames than what the model was trained on4_motion_inbetweening.ipynb
- A notebook experimenting with inbetweening between two motions using the inbetween way5_motion_blending.ipynb
- A notebook experimenting with blending between transition of two motions using the blending way
For example, we make a model learn a walking motion.
Walking.motion.-.Made.with.Clipchamp.1.mp4
Then we give it a constraint that it will need to hold a box, it should still know how to walk with the box in hand.
Walking.motion.with.box.-.Made.with.Clipchamp.mp4
For example, we make a model learn how to do a cartwheel
Cartwheel.-.Made.with.Clipchamp.1.mp4
Will it be able to do an aerial if we limit its usage of its hands?
Aeriel.-.Made.with.Clipchamp.mp4
These mujoco simulations were not done under a physics simulator, so its not physically accurate, the main goal is just to check if we can make a model retain its learning of a skill under a different environment
This library clashes with the other packages for the
pip install denoising_diffusion_pytorch
The setup instructions are exactly the same as described in the MDM repository for the environment and packages. See the MDM branch for more details
Setup the conda environment with the following command:
conda env create -f environment.yml
conda activate diffmimic
All the relevant code is kept within temporal_unet_diffusion.ipynb
. Headers and comments are left in the notebook to explain what each cell does.
One key thing to mention about motion datasets, is that the number of frames we load must be a multiple of 8 because of how the U-Net works. So if you have a spinkick motion with 78 frames, we can only load 72 frames. The motion_dataset
loader automatically chooses the maximum number of frames for you.
There are a few experiments already under the logs
directory.
I have included a script to play the generated motions. To use it, run the following command:
python3 mocap_player.py logs/{exp_name}/sampled_motions/motion1.npy
eg: python3 mocap_player.py logs/walk-motion/sampled_motions/motion1.npy
We can apply conditioning directly in the models/sampling_config.py
file under the apply_conditioning
function.
- There are 2 versions of the
apply_conditioning
function, one is the identity function and the other is the actual conditioning function. This is just an easy way to turn conditioning on and off. - Right now its just hardcoded to change the joint positions to look like the human model is holding a box. But essentially you can change these position tensors to anything you want.
In the joint configs, index 13-15 and 17-19 are shoulders representing an euler tuple, 16 and 20 are elbows representing a scalar rotation value in radians.
View original motion data, all of the original motion mocaps are stored in the diffusion/data/motions
directory. There is no CLI command to play the file, but you run the code in the following file to select a mocap and play it
python3 utils/mocap_v2.py
View sampled motion, all sampled motion are stored in the diffusion/logs
directory, under {experiment_name}/sampled_motions
. You can play these sampled motions using the following command
python3 mocap_player.py logs/test-constrained-sampling-holding-a-box/sampled_motions/motion1.npy