shimingliangdm / monocular-depth-unity

Depth from Monocular Image using the MiDaS v2 library with Unity's Barracuda inference framework

monocular-depth-unity

Real-time Depth from Monocular Image using the MiDaS v2 neural network with Unity's Barracuda inference framework.

This project includes the correct .onnx model that works well with Barracuda. You can always re-download the model from its official source if you encounter the infamous InvalidProtocolBufferException error.

See discussions that led to the choice of this specific model here (Unity) and here (Intel ISL). (June 2021)

Requirements

Platform	Version
Unity	2021.1
com.unity.barracuda	2.1.0-preview

Installation

Unity Package Manager

Add this line to your manifest.json:

"ulc-nn-depth":"https://github.com/GeorgeAdamon/monocular-depth-unity.git?path=/MonocularDepthBarracuda/Packages/DepthFromImage#main",

Usage

Step 0

Find the DEPTH_FROM_IMAGE prefab

Step 1

Use the Texture you like in the Input Texture slot. Works with RenderTextures and Texture2D objects. Video is supported through RenderTextures.

Step 2

Parameterize the visual output in the Depth Mesher object. Use Shader method for best performance, or Mesh to get an actual tangible mesh. If Color Texture is left blank, the mesh will be colorized with the depth data by default.

Performance

Sustained 60 fps on GTX 970 & i7 5930K (2015 rig) when using the shader-based displacement.

Issues

The mesh-based displacement doesn't fully utilize the AsyncGPUReadback command. A command queue needs to be implemented, to process pending commands.