vladmandic / human-bjs-vrm

Realtime VRM Humanoid Avatar Animation using Human Library and BabylonJS

Human: 3D Detection to VRM Avatar

WIP!!!

• Input
  Video file
  Live webcam
• Analyze and track all keypoints in a web worker thread
  Using Human library
  Face, Body, Hands
• Display
  Video Overlay with keypoints
  VRM Avatar with pan/zoom/rotate functionality

Run

Online

Live Demo

Offline

• Clone or fork the repository
• Install dependencies: npm install
• Start HTTPS server using precompiled code: npm start
• Navigate to https://localhost:8001

Development Mode

• Clone or fork the repository
• Compile code on-demand & start HTTPS server: npm run dev
• Navigate to https://localhost:8001

Notes

• All heavy-lifting (machine learning processing) is performed using Human library
• 2D visualization is performed using built-in draw methods in Human library
• 3D visualization is performed using BabylonJS

Processing

Process flow is separated into two loops:

• Process loop
  Runs in web-worker so its not blocking the main thread
  Supports WASM, WebGL and WebGPU backends
  Runs as fast as possible: frame is submited to worker and as soon as results are available a new frame is submitted
• Render loop
  Runs at fixed 60 FPS
  Draws both 2D canvas overlay and 3D GL mesh
  Uses intepolated (for smoother output) results based on last known processed results from process loop