The addon drives a 3D cartoon character by transferring SMPL's pose and global translation into the skeleton of the 3D character. Poses and global translation can be obtained from RGB images using ROMP or any SMPL-based 3D pose estimation model. If the estimation model outputs pose and global position at a high speed, then you can achieve the effect of driving 3D characters in real time in Blender.
Steps:
- Modify the path of results.npz in server.py and start server.py in the command line.
- Open Beta.blend or SMPL.blend and click on the triangle in the upper right corner.If a message is displayed in the lower left corner, the addon is running successfully.
- Go back to the Layout view and click on the small icon in the upper right to get the Texture.
- Press Ctrl+E to run the addon. At this time, the keyframe that is being transferred is displayed in the command line running server.py.
- Press the space in Blender to view the character animation.
If you want to use your own video, you need to use ROMP's video.sh to get a npz file.
Steps:
- Connect the camera and run ROMP's webcam_blender.sh.
- The rest of the steps are the same as steps 2, 3 of
From Video to Animation, where the 3D characters in Blender are driven in real time by pressing Ctrl+E.Press A to stop it.
If you are familiar with Blender and you want to use your own 3D character, make sure the base pose is T-pose and every joint's intial orientation is the identity matrix. In order to drive the character, the bones need to be named in the same way as the variables in the figure.
If you need a video background in the demo, select Compositing in the top menu bar, click Open Clip in the Movie Clip, and select your video. It is best not to use video as a background when driving in real time.
- 3D character is downloaded from Mixamo
- neuron_mocap_live-blender
- QuickMocap-BlenderAddon