Shopify / handy

The easiest way to mocap your hands!

handy

The easiest way to mocap your hands!

Background

No artists had to suffer to make this Blender animation. It was all mocapped using a Meta Quest Pro!

We developed this tool in order to streamline the process of capturing hand movements from Meta Quest headsets and bringing them into Blender for use in animations.

Examples of What You Can Do With This Tool

• The hands in this concept video were mocapped using handy!

• The hands and face in this live demo were also mocapped using handy!

The code behind that live demo is available here, in case you are interested in how mocap data can be played in the browser.

Prerequisites

• Unity 2022.1.23 or later
• Meta Quest (1, 2, or Pro)
• Windows or macOS

If you've never built and installed a Unity project on a Meta Quest headset, start by reading the beginner's guide. It will walk you through everything you need to do!

We wrote it so that people that don't have a technical background can run this project too.

Workflows

We developed two workflows for this project. The Optimized Workflow is fast, but it requires an internet connection so that the Handy app can send mocap recordings from the headset to the Unity editor to automatically generate Alembic files. The Manual Workflow is slow, but it doesn't require an internet connection.

Optimized Workflow

Click to expand

1. First, build the ClientScene and install it to the headset. You can find it here:

To build and install it you can simply go to File -> Build And Run.

2. Now, switch over to the ServerScene - double-click it to open it, then hit play in the editor. You can find the scene here:

Play in the editor looks like this:

3. Run the Handy app on the headset.

4. Start and stop recording by pinching your left thumb and index finger together and holding the pinch until the red recording indicator appears or disappears at your left wrist.

The red sphere at the left wrist indicates whether you are recording or not.

5. Every time you stop recording, the client (the Handy app) will send a .jsonlines file to the server (the Unity editor). The server will then immediately start playing back the recording (you will see your hands moving in Unity's viewport). Once it finishes playing the recording (you will see your hands freeze in Unity's viewport), it will output a finished .abc filename in Unity's console.

The .abc files will always be placed in a folder called data_output which is located at the root of your clone of this repository.

6. If you start and stop recording multiple times in a row, a queue will form in the server, so you will have to wait for it to process all your recordings.

7. Load your exported .abc files in Blender. You will see two hands and a cube that acts as a placeholder for the headset, which we also record!

Manual Workflow

Click to expand

1. First, build the CaptureScene and install it to the headset. You can find it here:

Double-click it to open it, and then go to File -> Build Settings..., select any scenes that are in the Scenes In Build box, right click them and select Remove Selection.

After that simply click the Add Open Scenes button and the CaptureScene should be added to the list. You are now ready to press the Build And Run button.

2. Run the Handy app on the headset.
3. Start and stop recording by pinching your left thumb and index finger together and holding the pinch until the red recording indicator appears or disappears at your left wrist.

The red sphere at the left wrist indicates whether you are recording or not.

4. Connect the headset to your computer and download the .jsonlines files that were recorded. You can find them here:

5. Open the PlaybackScene in the Unity editor by double-clicking it. You can find it here:

6. Click on the PlaybackManager object in the scene hierarchy. In the PlaybackManager component of that object, input the path of the .jsonlines file that you want to export as an Alembic file. In the AlembicExporter component of the same object, specify the location where you want the Alembic file to be generated and its name.

7. Hit play in the editor and wait for the animation to complete. Play in the editor looks like this:

8. Load your exported .abc file in Blender. You will see two hands and a cube that acts as a placeholder for the headset, which we also record!

License

This project is licensed under the MIT License - see the LICENSE file for details.