imclab / ARFoundationReplay

AR Record and Playback System on Unity ARFoundation

AR Foundation Replay

🚧 Work in progress 🏗️

PoC of Recording AR Foundation session into a single mp4 with a binary timeline track / Replaying it in Editor on AR Foundation without any extra settings.

2023-10-10-walking-geospatial.mp4

↓ Check out the complete project WorldEnsambe utilizing ARFoundationReplay.

editor-simulation.mp4

Tested platform

• Only iOS ARKit + macOS Editor for now. Android ARCore is planned.
• Tested with the latest Unity2022 LTS.
• Supported only URP. Built-in is not supported for now.

Implemented Subsystems

• XRSessionSubsystem
• XRCameraSubsystem
• XRPointCloudSubsystem
• XRPlaneSubsystem
• XRAnchorSubsystem
• XRRaycastSubsystem
• XRHumanBodySubsystem
• XREnvironmentProbeSubsystem
• XRInputSubsystem
• XRImageTrackingSubsystem
• XRObjectTrackingSubsystem
• XRFaceSubsystem
• XROcclusionSubsystem
• XRParticipantSubsystem
• XRMeshSubsystem
• ARCore Extensions
  • Earth Manager
  • Cloud Anchors
  • Streetscape Geometry

How to use

• Download System.Runtime.CompilerServices.Unsafe NuGet package and rename to zip to extract. Locate lib/netstandard2.0/System.Runtime.CompilerServices.Unsafe.dll to Assets/Plugins/ directory. Or you can copy it from this repository.

• Add the following line to Packages/manifest.json to install AR Foundation Replay package via UPM:

  "dependencies": {
      "com.cysharp.memorypack": "https://github.com/Cysharp/MemoryPack.git?path=src/MemoryPack.Unity/Assets/Plugins/MemoryPack#1.9.16",
      "com.github.asus4.arfoundationreplay": "https://github.com/asus4/ARFoundationReplay.git?path=Packages/com.github.asus4.arfoundationreplay",
      ... other dependencies
  }

• Simulation on the Unity Editor is supported only on iOS. Open Build Settings and switch the platform to iOS.

• Activate AR Foundation Replay for Unity Editor:

  1. Open Project Settings/XR Plug-in Management.
  2. Select the PC tab in the XR Plug-in Management.
  3. Activate AR Foundation Replay

• Put the ARRecordButton prefab into the Scene which is located at Packages/com.github.asus4.arfoundationreplay/Prefabs/ARRecordButton.prefab.

• Record AR on the device.

• You can change the replay file from Project Settings/XR Plug-in Management/AR Foundation Replay.

• Play the AR in the Editor.

ARCore Extensions Support

It supports replaying ARCore Geospatial. To test it, please use the forked version of the arcore-unity-extensions and refer to the ARFoundationReplayGeospatial example project.

Dependencies

• MemoryPack - MIT License
• XR SDK - Unity Companion License
• keijiro/Bibcam - Unlicense
• keijiro/Avfi - Unlicense: Metadata recording system is modified from Avfi.