DufeRob / mediapipe-touchdesigner

GPU Accelerated MediaPipe Plugin for TouchDesigner

MediaPipe TouchDesigner Plugin

A GPU Accelerated, self-contained, MediaPipe Plugin for TouchDesigner that runs on Mac and PC with no installation. This project currently supports all MediaPipe vision models except Interactive Segmentation and Image Embedding.

Setup

Download the latest release.zip from the Release Section. Open up the MediaPipe TouchDesigner.toe file. All of the components are stored inside the /toxes folder. The main component is MediaPipe.tox. All of the other components are examples of how to load and display the associated model data in TouchDesigner.

On the MediaPipe component once it's loaded you can select your webcam from the drop-down. You can turn on and off the different MediaPipe models as well as preview overlays. There're also sub-menus available for each model to customize them further.

A note on resolution

Currently the model is limited to 720p input resolution, but we're working on higher resolutions

Components

The plugin consists of a number of components:

Media Pipe tox

This component launches a Chromium web browser to host and run all of the MediaPipe vision tasks. It has a DAT output for each vision task, as well as a TOP output showing the video feed and any overlays.

Face detector tox

Use this to process the face detection results

Face detection guide

Face tracking tox

Use this to process the facial landmark detection results

Face landmark detection guide

Hand tracking tox

Use this to process the hand landmark and gesture detection results

Gesture recognition task guide

Object tracking tox

Use this to process the object detection results

Object detection task guide

Pose tracking tox

Use this to process the pose landmark detection results

Pose landmark detection guide

Image segmentation tox

Use this to key out segmentation results Image segmentation guide

Note: If you're hoping to get the most accurate web-cam cutout use the MultiCam model in the MediaPipe.tox. There's also a toggle to display only the background cutout which you can enable while on the multiclass model.

Image classification tox

Use this to identify what the image might contain Image classification guide

Sending TOPs from TouchDesigner

SpoutCam on Windows

If you are on Windows, you can use Spout to send any TOP from TouchDesigner to MediaPipe by using a Syphon Spout Out TOP and SpoutCam.

Download SpoutCam

SpoutCam

Unzip the folder

Run SpoutCam Settings.exe

Note there is no installer, it will run from wherever the folder is

• Enter the frame rate and resolution to match your source material in TouchDesigner
• Enter the default TouchDesigner Spout output name of TDSyphonSpoutOut into the Starting Sender box
• Click register to create the virtual webcam

Drop a Syphon Spout Out TOP into your project

Feed your desired TOP into it

In MediaPipe, select SpoutCam as your webcam source

The media should appear with no frames of delay!

Syphon on Mac

There is no SpoutCam equivalent on Mac, but you can use Syphon to send video to OBS, and then use the OBS Virtual Webcam output to the MediaPipe task. It's not the most elegant solution, but it works.

Performance tips

Realtime CHOP

There are lots of interesting bits of data in the CHOP output of the MediaPipe tox:

detectTime

How long the MP detector took to run in ms.

drawTime

How long the overlays and segmentation colors took to draw in ms.

sourceFrameRate

The frame rate of the webcam video source MediaPipe is using.

realTimeRatio

What ratio of a frame it took to process the video. So 0.1 means 0.1 of a frame was needed for MediaPipe to process. This uses the project's cookRate as a reference

totalInToOutDelay

The number of project frames delay MediaPipe introduces. In TouchDesigner 2022.33910 this is at least 3 frames for the web browser component. If you are using Spout or Syphon to send video to a virtual webcam, you can use this as a parameter in a cache TOP to sync your input video source back up with the MediaPipe output TOP.

isRealTime

Tells you if the whole process is able to keep up with the input frame rate.

Turn off what you're not using

The MediaPipe detection tasks are very CPU and GPU intensive, so turn off any that you aren't using

Hyperthreading

If you are on a PC, you can greatly improve all CPU-based render times within TouchDesigner, including the MediaPipe tasks by disabling HyperThreading (Intel CPUs) or Simultaneous Multi-Threading (SMT - AMD CPUs). On my laptop it was a 60-80% improvement.

Enabling/disabling HyperThreading is done in your system BIOS, so look up the instructions on how to do that for your computer. You can re-enable HyperThreading at any point if you want to go back and it is not a risky change to make to your system.

How do the plugins work?

This project loads the different MediaPipe models through a web browser. All of the ML models are downloaded locally and stored inside TouchDesigner's virtual file system including the website so the component can run without an internet connection. The models run using web assembly and the data coming back from the models are piped into TouchDesigner through a local WebSocket server running inside TD. This allows the components to run as standalone .tox files with GPU acceleration on any device with no setup.

Building from source

This package uses yarn with vite inside node, which gives you a few options. Firstly, you need to download and install node.js

Installation

yarn install

This installs vite and all the other dependencies required. Should only be needed the first time.

Debugging of the web page

yarn dev

This launches a tiny web server on port 5173 that fires a reload to the browser every time you save a file change. Super useful for debugging the web page.

Suggested use for this:

1. Run yarn dev via the command line, or the console window of VSCode
2. Load the TouchDesigner project and toggle all the settings you'll want to test with.
3. Enter the MediaPipe COMP
4. Disable the webbrowser component by clicking the X
5. Copy the current_url DAT string
6. Paste the url into a new Chrome tab
7. Edit the url and replace the port number with 5173, so for example replace localhost:3001 with localhost:5173

Chrome will now load the page from yarn dev, while still communicating with TouchDesigner via websockets. Note that parameter changes you make in TouchDesigner will not be reflected in the web page unless you repeat the steps to copy the url.

Build

yarn build

This builds all the web page files and puts them into the _mpdist folder. If this folder exists, the web pages used within the MediaPipe COMP will be served from here.

Version Update / Release

There is a build_release COMP in the top level at the top of the toe file that handles an automated build process.

To create a release zip file:

1. Open the MediaPipe TouchDesigner.toe file
2. Navigate to the layout you want to be loaded when someone opens the project for the first time.
3. Press Ctrl Alt B to trigger a build. The project will lock up for a while during the build process.
4. The build script will:

• Remove and recreate a release folder.
• Perform a yarn install
• Perform a yarn build
• Load all files from _mpdist into the MediaPipe COMP Virtual File System
• Remove all external script references on DATs
• Export all tox files to release/toxes
• Save the toe file to release
• Zip the contents of the release folder to release.zip
• Give you a popup showing the number of errors.
• Clicking ok (or anywhere!) causes the toe file to reload from the main folder so you're back where you started.
• If there were any errors in the build, you can turn on the TouchDesigner text console and run the build again to see what the errors are.