qiaow02 / MediaPipeUnityPlugin

Unity plugin to run MediaPipe graphs

MediaPipe Unity Plugin

This is a Unity (2021.3.3f1) Native Plugin to use MediaPipe (0.8.10).

The goal of this project is to port the MediaPipe API (C++) one by one to C# so that it can be called from Unity.
This approach may sacrifice performance when you need to call multiple APIs in a loop, but it gives you the flexibility to use MediaPipe instead.

With this plugin, you can

• Write MediaPipe code in C#.
• Run MediaPipe's official solution on Unity.
• Run your custom Calculator and CalculatorGraph on Unity.
  • ⚠️ Depending on the type of input/output, you may need to write C++ code.

😸 Hello World!

Here is a Hello World! example.
Compare it with the official code!

using Mediapipe;
using UnityEngine;

public sealed class HelloWorld : MonoBehaviour
{
    private const string _ConfigText = @"
input_stream: ""in""
output_stream: ""out""
node {
  calculator: ""PassThroughCalculator""
  input_stream: ""in""
  output_stream: ""out1""
}
node {
  calculator: ""PassThroughCalculator""
  input_stream: ""out1""
  output_stream: ""out""
}
";

    private void Start()
    {
        var graph = new CalculatorGraph(_ConfigText);
        var poller = graph.AddOutputStreamPoller<string>("out").Value();
        graph.StartRun().AssertOk();

        for (var i = 0; i < 10; i++)
        {
            graph.AddPacketToInputStream("in", new StringPacket("Hello World!", new Timestamp(i))).AssertOk();
        }

        graph.CloseInputStream("in").AssertOk();
        var packet = new StringPacket();

        while (poller.Next(packet))
        {
            Debug.Log(packet.Get());
        }
        graph.WaitUntilDone().AssertOk();
    }
}

For more detailed usage, see the API Overview page or the tutorial on the Getting Started page.

🛠️ Installation

This repository does not contain required libraries (e.g. libmediapipe_c.so, Google.Protobuf.dll, etc), so you need to build them first.
For step-by-step guide, please refer to the Installation Guide in Wiki.

⚠️ libraries that can be built differ depending on your environment.

Supported Platforms

⚠️ GPU mode is not supported on macOS and Windows.

	Editor	Linux (x86_64)	macOS (x86_64)	macOS (ARM64)	Windows (x86_64)	Android	iOS	WebGL
Linux (AMD64) 1	✔️	✔️				✔️
Intel Mac	✔️		✔️			✔️	✔️
M1 Mac 2	✔️			✔️		✔️	✔️
Windows 10/11 (AMD64) 3	✔️				✔️	✔️

🍽️ Try sample app

Example Solutions

Here is a list of solutions that you can try in the sample app.

🔔 The graphs you can run are not limited to the ones in this list.

	Android	iOS	Linux (GPU)	Linux (CPU)	macOS (CPU)	Windows (CPU)	WebGL
Face Detection	✔️	✔️	✔️	✔️	✔️	✔️
Face Mesh	✔️	✔️	✔️	✔️	✔️	✔️
Iris	✔️	✔️	✔️	✔️	✔️	✔️
Hands	✔️	✔️	✔️	✔️	✔️	✔️
Pose	✔️	✔️	✔️	✔️	✔️	✔️
Holistic	✔️	✔️	✔️	✔️	✔️	✔️
Selfie Segmentation	✔️	✔️	✔️	✔️	✔️	✔️
Hair Segmentation	✔️	✔️	✔️	✔️	✔️	✔️
Object Detection	✔️	✔️	✔️	✔️	✔️	✔️
Box Tracking	✔️	✔️	✔️	✔️	✔️	✔️
Instant Motion Tracking	✔️	✔️	✔️	✔️	✔️	✔️
Objectron	✔️	✔️	✔️	✔️	✔️	✔️
KNIFT

UnityEditor

Select Mediapipe/Samples/Scenes/Start Scene and play.

Desktop

If you've built native libraries for CPU (i.e. --desktop cpu), select CPU for inference mode from the Inspector Window.

Android, iOS

Make sure that you select GPU for inference mode before building the app, because CPU inference mode is not supported currently.

📖 Wiki

https://github.com/homuler/MediaPipeUnityPlugin/wiki

📜 LICENSE

MIT

Note that some files are distributed under other licenses.

• MediaPipe (Apache Licence 2.0)
• emscripten (MIT)
  • third_party/mediapipe_emscripten_patch.diff contains code copied from emscripten
• FontAwesome (LICENSE)
  • Sample scenes use Font Awesome fonts

See also Third Party Notices.md.

Footnotes

1. Tested on Arch Linux. ↩

2. Experimental, because MediaPipe does not support M1 Mac. ↩

3. Running MediaPipe on Windows is experimental. ↩