langyo / SysMocap

A real-time motion capture system for 3D virtual character animating.

SysMocap

English Version | 中文版本

A cross-platform real-time video-driven motion capture and 3D virtual character rendering system for VTuber/Live/AR/VR.

Available for Windows, macOS (packaged) & Linux (source code only)

Download Now (zipped, without installation)

(This is a multi-language software, including English. )

Highlights

🌟 Beautiful GUI with Material Design 3 Color System, with Dark Mode supported.

🌟 Easy to used. You can import 3D models just with drags.

add-model-drag.mp4

🌟 Support WebXR API on Mocap Forwarding (HTTPS only)

webxr-ar-demo.mp4

🌟 Model viewer with bones & dressing controller

🌟 Support OBS live-streaming

🌟 Support full-body motion capture

🌟 Support Auto Skeleton Detection for All VRM files and Mixamo Format FBX files

🌟 Support Any Skeleton Structure with Manual Mapping

🌟 Does not require a discrete graphics card and runs smoothly even on eight-year-old computers (i7-4790k/GTX770/16G RAM)

🌟 Powered by Mediapipe and Kalidokit with Web Technologies

More Effect Demonstration

🌟 Facial

🌟 Half-body

🌟 Half-body with Hands

🌟 Full-body

System architecture

How to use

Run on your computer from source code (need lastest Node.js):

git clone https://github.com/xianfei/SysMocap.git
cd SysMocap
npm i
npm start

Bugs

• You tell me

Notice

1. HTTP & HTTPS will use same port in Mocap Data Forward.

Required Skeleton Node in glTF/glb/FBX Model File for Macap:

(If not same as above, you need rebind them manually.)

• Hips (Main Node, both Position and Rotation. Ratation only for other nodes)

• Neck

• Chest

• Spine

• RightUpperArm

• RightLowerArm

• LeftUpperArm

• LeftLowerArm

• LeftUpperLeg

• LeftLowerLeg

• RightUpperLeg

• RightLowerLeg

Star History

Development progress

To-Do

• Settings page and global settings utils

• Add play/pause button and progress bar when mocap from video

• Support bones binding for glTF/glb

• Support rendering glTF/glb model

• Support binding when bones' name is non-uniformed

• Model library add user's custom 3D model

• Live plug-in / interface for Open Broadcast Software

• Output video ( using such as libffmpeg )

• Support per-frame rendering without drop frame

• Support c-s architecture for online video mocap ( on cloud )

• Support Material Designed 3 Color System (color picking)

• Mocap data forwarding via network

• Adapt for Linux and macOS

Thanks

You can see all dependencies in package.json

• google/mediapipe/Holistic for Mocap

• kalidokit for Calulate Mocap Data

• electron and Vue.js for GUI Framework

• Material color utilities for Color Picking

Cite

@INPROCEEDINGS{9974484,
  author={Song, Wenfeng and Wang, Xianfei and Gao, Yang and Hao, Aimin and Hou, Xia},
  booktitle={2022 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)}, 
  title={Real-time Expressive Avatar Animation Generation based on Monocular Videos}, 
  year={2022},
  volume={},
  number={},
  pages={429-434},
  doi={10.1109/ISMAR-Adjunct57072.2022.00092}}