An Unreal Engine plugin that helps you use AI and ML techniques in your unreal engine project.

Introduction

MediaPipe4U provides a suite of libraries and tools for you to quickly apply artificial intelligence (AI) and machine learning (ML) techniques in Unreal Engine project. You can plug these solutions into your UE project immediately, customize them to your needs. Include motion capture , facial expression capture for your 3D avatar, text to speech (TTS), speech recorgnization (ASR), etc. All features are real-time, offline, low latency and easy to use.

What's news

Latest update: 20231010

[new] 🌈 MediaPipe4U Mobile Android packaging support, the development process is fully compatible with Windows.
[new] 🔥 Both StartCamera and StartImageSource now support truly blueprint-friendly asynchronous functions.
[improve] Optimize frame pipline, huge performance improve, now Windows CPU can reach 25 fps (CPU: AMD 3600, Frame Size: 1280 * 720).
[breaking change]:rotating_light:Many functions are now BlueprintPure and may require changes to your blueprint.

Release Notes

For the release notes, ref below:

Features

💚All features are pure C++, no Python or external programs required.

Motion Capture
- Motion of the body
- Motion of the fingers
- Movement
- Drive 3D avatar
- Real-time
- RGB webcam supported
Face Capture
- Facial expression.
- Arkit Blendshape compatible (52 expression)
- Live link compatible
- Real-time
- RGB webcam supported
Multi-source Capture
- RGB WebCam
- Video File
- Image
- Living Stream (RTMP/SMTP)
- Android Device (M4U Remoting)
LLM
- Offline
- CPU/GPU Inference
- Multiple models
  - LLaMA/LLaMA2
  - ChatGLM (work in progress)
TTS
- Offline
- Real-time
- Lip-Sync
- Multiple models
  - Paddle Speech: Chinese, English
  - Bark: 13 languages (work in progress)
ASR
- Offline
- Real-time
- Multiple models
  - FunASR: Chinese
  - Whisper: 99 languages
Animation Data Export
- BVH export
Pure plugins
- No external programs required
- All in Unreal Engine

Document

Download

Unreal Engine	China Site	Global Site	Update
UE 5.0	奶牛快传	One Drive	2023-10-10
UE 5.1	奶牛快传	One Drive	2023-10-10
UE 5.2	奶牛快传	One Drive	2023-10-10

Because the plugin is precompiled and contains a large number of C++ link symbols and debug symbols, it will cost 10G disk space after decompression (most files are UE-generated binaries in Intermediate).
Don't need to worry about disk usage, this is just disk usage during development, after the project is packaged, the plug-in disk usage is 300M only (most files are GStreamer dynamic library and speech models).

Cross Platform

Now, M4U support Android and Windows (Linux is coming soom)

Plugins (Modules)	Windows	Android	Linux
MediaPipe4U	✔️	✔️	Coming Soon
MediaPipe4ULiveLink	✔️	✔️	Coming Soon
GStreamer	✔️	❌	Coming Soon
MediaPipe4UGStreamer	✔️	❌	Coming Soon
MediaPipe4UBVH	✔️	❌	Coming Soon
MediaPipe4USpeech	✔️	❌	Coming Soon
MediaPipe4ULLM	✔️	❌	Coming Soon

Free License File Release

The license file will be published in the discussion, and the plugin package file will automatically include an license file.

M4U Remoting App

Android Version	Download Link	Update
Android 7.0 or later	Download	2023-04-21

About M4U Remoting

Note
This is a commercial license exclusive feature: capturing facial expressions from android device.
Free license only supports using in UE Editor, cannot be packaged this feature.
M4U Remoting Document

Demo Project

Please clone this repository to get demo project:

https://github.com/endink/MediaPipe4U-Demo

The demo project does not contain plugins, you need to download the plugin and copy content to the project's plugins folder to run.

Video Channels

Video Tutorials (English)

Video Tutorials (Chinese)

bilibili

FAQ

If you have any questiongs, please check FAQ first. The problems listed there may be also yours. If you can’t find an answer in the FAQ, please post an issue. Private message or emal may cause the question to be mised .

Performance

Since the Windows version of MediaPipe does not support GPU inference, Windows relies on the CPU to inferring human pose estimation (see MediaPipe offical site for more details).

Evaluation

Frame Rate： 18-24 fps

CPU usage：20% (Based on DEMO project)

Testing Evnrioment

CPU： AMD 3600 CPU
RAM: 32GB
GPU: Nvidia 1660s

Follow Me