feixuwu/semu.xr.openxr

OpenXR compact binding for creating extended reality applications on NVIDIA Omniverse

This extension provides a compact python binding (on top of the open standard OpenXR for augmented reality (AR) and virtual reality (VR)) to create extended reality applications taking advantage of NVIDIA Omniverse rendering capabilities. In addition to updating views (e.g., head-mounted display), it enables subscription to any input event (e.g., controller buttons and triggers) and execution of output actions (e.g., haptic vibration) through a simple and efficient API for accessing conformant devices such as HTC Vive, Oculus and others...

Target applications: Any NVIDIA Omniverse app with the omni.syntheticdata extension installed (e.g., Isaac Sim, Code, etc.)

Tested in Ubuntu 18.04/20.04, STEAMVR beta 1.24.3, Omniverse Code 2022.1.3 and Isaac Sim 2022.1.1

Supported OS: Linux

Changelog: CHANGELOG.md

Table of Contents:

Extension setup
Diagrams
Sample code
GUI launcher
Extension API

Extension setup

Add the extension using the Extension Manager or by following the steps in Extension Search Paths
- Git url (git+https) as extension search path
```
git+https://github.com/Toni-SM/semu.xr.openxr.git?branch=main&dir=exts
```
- Compressed (.zip) file for import
  
  semu.xr.openxr.zip
Enable the extension using the Extension Manager or by following the steps in Extension Enabling/Disabling
Import the extension into any python code and use it...
```
from semu.xr.openxr import _openxr
```
Or use the GUI launcher to directly dislpay the current stage in the HMD

Diagrams

High-level overview of extension usage, including the order of function calls, callbacks and the action and rendering loop

Typical OpenXR application showing the grouping of the standard functions under the compact binding provided by the extension (adapted from openxr-10-reference-guide.pdf)

Sample code

The following sample code shows a typical workflow that configures and renders on a stereo headset the view generated in an Omniverse application. It configures and subscribes two input actions to the left controller to 1) mirror on a simulated sphere the pose of the controller and 2) change the dimensions of the sphere based on the position of the trigger. In addition, an output action, a haptic vibration, is configured and executed when the controller trigger reaches its maximum position

A short video, after the code, shows a test of the OpenXR application from the Script Editor using an HTC Vive Pro

import omni
from pxr import UsdGeom
from semu.xr.openxr import _openxr

# get stage unit
stage = omni.usd.get_context().get_stage()
meters_per_unit = UsdGeom.GetStageMetersPerUnit(stage)

# create a sphere (1 centimeter radius) to mirror the controller's pose
sphere_prim = omni.usd.get_context().get_stage().DefinePrim("/sphere", "Sphere")
sphere_prim.GetAttribute("radius").Set(0.01 / meters_per_unit)

# acquire interface
xr = _openxr.acquire_openxr_interface()

# setup OpenXR application using default parameters
xr.init()
xr.create_instance()
xr.get_system()

# action callback
def on_action_event(path, value):
    # process controller's trigger
    if path == "/user/hand/left/input/trigger/value":
      # modify the sphere's radius (from 1 to 10 centimeters) according to the controller's trigger position
      sphere_prim.GetAttribute("radius").Set((value * 9 + 1) * 0.01 / meters_per_unit)
      # apply haptic vibration when the controller's trigger is fully depressed
      if value == 1:
        xr.apply_haptic_feedback("/user/hand/left/output/haptic", {"duration": _openxr.XR_MIN_HAPTIC_DURATION})
    # mirror the controller's pose on the sphere (cartesian position and rotation as quaternion)
    elif path == "/user/hand/left/input/grip/pose":
        xr.teleport_prim(sphere_prim, value[0], value[1])

# subscribe controller actions (haptic actions don't require callbacks) 
xr.subscribe_action_event("/user/hand/left/input/grip/pose", callback=on_action_event, reference_space=_openxr.XR_REFERENCE_SPACE_TYPE_LOCAL)
xr.subscribe_action_event("/user/hand/left/input/trigger/value", callback=on_action_event)
xr.subscribe_action_event("/user/hand/left/output/haptic")

# create session and define interaction profiles
xr.create_session()

# setup cameras and viewports and prepare rendering using the internal callback
xr.set_meters_per_unit(meters_per_unit)
xr.setup_stereo_view()
xr.set_frame_transformations(flip=0)
xr.set_stereo_rectification(y=0.05)

# execute action and rendering loop on each simulation step
def on_simulation_step(step):
    if xr.poll_events() and xr.is_session_running():
        xr.poll_actions()
        xr.render_views(_openxr.XR_REFERENCE_SPACE_TYPE_LOCAL)

physx_subs = omni.physx.get_physx_interface().subscribe_physics_step_events(on_simulation_step)

snippet.mp4

GUI launcher

The extension also provides a graphical user interface that helps to launch a partially configurable OpenXR application form a window. This interface is located in the Add-ons > OpenXR UI menu

The first four options (Graphics API, Form factor, Blend mode, View configuration type) cannot be modified once the OpenXR application is running. They are used to create and configure the OpenXR instance, system and session

The other options (under the central separator) can be modified while the application is running. They help to modify the pose of the reference system, or to perform transformations on the images to be rendered, for example.

Extension API

Acquiring extension interface

Acquire OpenXR interface

_openxr.acquire_openxr_interface() -> semu::xr::openxr::OpenXR

Release OpenXR interface

_openxr.release_openxr_interface(xr: semu::xr::openxr::OpenXR) -> None

API

The following functions are provided on the OpenXR interface:

Init OpenXR application by loading the necessary libraries
```
init(graphics: str = "OpenGL", use_ctypes: bool = False) -> bool
```
Parameters:
- graphics: str
  
  OpenXR graphics API supported by the runtime (OpenGL, OpenGLES, Vulkan, D3D11, D3D12). Note: At the moment only OpenGL is available
- use_ctypes: bool, optional
  
  If true, use ctypes as C/C++ interface instead of pybind11 (default)
Returns:
- bool
  
  True if initialization was successful, otherwise False

Get OpenXR session's running status
```
is_session_running() -> bool
```
Returns:
- bool
  
  Return True if the OpenXR session is running, False otherwise

Create an OpenXR instance to allow communication with an OpenXR runtime
```
create_instance(application_name: str = "Omniverse (XR)", engine_name: str = "", api_layers: list = [], extensions: list = []) -> bool
```
Parameters:
- application_name: str, optional
  
  Name of the OpenXR application (default: Omniverse (XR))
- engine_name: str, optional
  
  Name of the engine (if any) used to create the application (empty by default)
- api_layers: list of str, optional
  
  API layers to be inserted between the OpenXR application and the runtime implementation
- extensions: list of str, optional
  
  Extensions to be loaded. Note: The graphics API selected during initialization (init) is automatically included in the extensions to be loaded. At the moment only the graphic extensions are configured
Returns:
- bool
  
  True if the instance has been created successfully, otherwise False

Obtain the system represented by a collection of related devices at runtime
```
get_system(form_factor: int = XR_FORM_FACTOR_HEAD_MOUNTED_DISPLAY, blend_mode: int = XR_ENVIRONMENT_BLEND_MODE_OPAQUE, view_configuration_type: int = XR_VIEW_CONFIGURATION_TYPE_PRIMARY_STEREO) -> bool
```
Parameters:
- form_factor: {XR_FORM_FACTOR_HEAD_MOUNTED_DISPLAY, XR_FORM_FACTOR_HANDHELD_DISPLAY}, optional
  
  Desired form factor from XrFormFactor enum (default: XR_FORM_FACTOR_HEAD_MOUNTED_DISPLAY)
- blend_mode: {XR_ENVIRONMENT_BLEND_MODE_OPAQUE, XR_ENVIRONMENT_BLEND_MODE_ADDITIVE, XR_ENVIRONMENT_BLEND_MODE_ALPHA_BLEND}, optional
  
  Desired environment blend mode from XrEnvironmentBlendMode enum (default: XR_ENVIRONMENT_BLEND_MODE_OPAQUE)
- view_configuration_type: {XR_VIEW_CONFIGURATION_TYPE_PRIMARY_MONO, XR_VIEW_CONFIGURATION_TYPE_PRIMARY_STEREO}, optional
  
  Primary view configuration type from XrViewConfigurationType enum (default: XR_VIEW_CONFIGURATION_TYPE_PRIMARY_STEREO)
Returns:
- bool
  
  True if the system has been obtained successfully, otherwise False

Create an OpenXR session that represents an application's intention to display XR content
```
create_session() -> bool
```
Returns:
- bool
  
  True if the session has been created successfully, otherwise False

Event polling and processing
```
poll_events() -> bool
```
Returns:
- bool
  
  False if the running session needs to end (due to the user closing or switching the application, etc.), otherwise False

Action polling
```
poll_actions() -> bool
```
Returns:
- bool
  
  True if there is no error during polling, otherwise False

Present rendered images to the user's views according to the selected reference space
```
render_views(reference_space: int = XR_REFERENCE_SPACE_TYPE_LOCAL) -> bool
```
Parameters:
- reference_space: {XR_REFERENCE_SPACE_TYPE_VIEW, XR_REFERENCE_SPACE_TYPE_LOCAL, XR_REFERENCE_SPACE_TYPE_STAGE}, optional
  
  Desired reference space type from XrReferenceSpaceType enum used to render the images (default: XR_REFERENCE_SPACE_TYPE_LOCAL)
Returns:
- bool
  
  True if there is no error during rendering, otherwise False

Create an action given a path and subscribe a callback function to the update event of this action

subscribe_action_event(path: str, callback: Union[Callable[[str, object], None], None] = None, action_type: Union[int, None] = None, reference_space: Union[int, None] = XR_REFERENCE_SPACE_TYPE_LOCAL) -> bool

If action_type is None the action type will be automatically defined by parsing the last segment of the path according to the following policy:

Action type (`XrActionType`)	Last segment of the path
`XR_ACTION_TYPE_BOOLEAN_INPUT`	/click, /touch
`XR_ACTION_TYPE_FLOAT_INPUT`	/value, /force
`XR_ACTION_TYPE_VECTOR2F_INPUT`	/x, /y
`XR_ACTION_TYPE_POSE_INPUT`	/pose
`XR_ACTION_TYPE_VIBRATION_OUTPUT`	/haptic, /haptic_left, /haptic_right, /haptic_left_trigger, /haptic_right_trigger

The callback function (a callable object) should have only the following 2 parameters:

path: str

The complete path (user path and subpath) of the action that invokes the callback

value: bool, float, tuple(float, float), tuple(pxr.Gf.Vec3d, pxr.Gf.Quatd)

The current state of the action according to its type

Action type (`XrActionType`)	python type
`XR_ACTION_TYPE_BOOLEAN_INPUT`	`bool`
`XR_ACTION_TYPE_FLOAT_INPUT`	`float`
`XR_ACTION_TYPE_VECTOR2F_INPUT` (x, y)	`tuple(float, float)`
`XR_ACTION_TYPE_POSE_INPUT` (position (in stage unit), rotation as quaternion)	`tuple(pxr.Gf.Vec3d, pxr.Gf.Quatd)`

XR_ACTION_TYPE_VIBRATION_OUTPUT actions will not invoke their callback function. In this case the callback must be None

XR_ACTION_TYPE_POSE_INPUT also specifies, through the definition of the reference_space parameter, the reference space used to retrieve the pose

The collection of available paths corresponds to the following interaction profiles:

Parameters:

path: str

Complete path (user path and subpath) referring to the action
callback: callable object (2 parameters) or None for XR_ACTION_TYPE_VIBRATION_OUTPUT

Callback invoked when the state of the action changes
action_type: {XR_ACTION_TYPE_BOOLEAN_INPUT, XR_ACTION_TYPE_FLOAT_INPUT, XR_ACTION_TYPE_VECTOR2F_INPUT, XR_ACTION_TYPE_POSE_INPUT, XR_ACTION_TYPE_VIBRATION_OUTPUT} or None, optional

Action type from XrActionType enum (default: None)
reference_space: {XR_REFERENCE_SPACE_TYPE_VIEW, XR_REFERENCE_SPACE_TYPE_LOCAL, XR_REFERENCE_SPACE_TYPE_STAGE}, optional

Desired reference space type from XrReferenceSpaceType enum used to retrieve the pose (default: XR_REFERENCE_SPACE_TYPE_LOCAL)

Returns

bool

True if there is no error during action creation, otherwise False

Apply a haptic feedback to a device defined by a path (user path and subpath)
```
apply_haptic_feedback(path: str, haptic_feedback: dict) -> bool
```
Parameters:
- path: str
  
  Complete path (user path and subpath) referring to the action
- haptic_feedback: dict
  
  A python dictionary containing the field names and value of a XrHapticBaseHeader-based structure. Note: At the moment the only haptics type supported is the unextended OpenXR XrHapticVibration
Returns:
- bool
  
  True if there is no error during the haptic feedback application, otherwise False

Stop a haptic feedback applied to a device defined by a path (user path and subpath)
```
stop_haptic_feedback(path: str) -> bool
```
Parameters:
- path: str
  
  Complete path (user path and subpath) referring to the action
Returns:
- bool
  
  True if there is no error during the haptic feedback stop, otherwise False

Setup Omniverse viewport and camera for monoscopic rendering
```
setup_mono_view(camera: Union[str, pxr.Sdf.Path, pxr.Usd.Prim] = "/OpenXR/Cameras/camera", camera_properties: dict = {"focalLength": 10}) -> None
```
This method obtains the viewport window for the given camera. If the viewport window does not exist, a new one is created and the camera is set as active. If the given camera does not exist, a new camera is created with the same path and set to the recommended resolution of the display device

Parameters:
- camera: str, pxr.Sdf.Path or pxr.Usd.Prim, optional
  
  Omniverse camera prim or path (default: /OpenXR/Cameras/camera)
- camera_properties: dict
  
  Dictionary containing the camera properties supported by the Omniverse kit to be set (default: {"focalLength": 10})

Setup Omniverse viewports and cameras for stereoscopic rendering
```
setup_stereo_view(left_camera: Union[str, pxr.Sdf.Path, pxr.Usd.Prim] = "/OpenXR/Cameras/left_camera", right_camera: Union[str, pxr.Sdf.Path, pxr.Usd.Prim, None] = "/OpenXR/Cameras/right_camera", camera_properties: dict = {"focalLength": 10}) -> None
```
This method obtains the viewport window for each camera. If the viewport window does not exist, a new one is created and the camera is set as active. If the given cameras do not exist, new cameras are created with the same path and set to the recommended resolution of the display device

Parameters:
- left_camera: str, pxr.Sdf.Path or pxr.Usd.Prim, optional
  
  Omniverse left camera prim or path (default: /OpenXR/Cameras/left_camera)
- right_camera: str, pxr.Sdf.Path or pxr.Usd.Prim, optional
  
  Omniverse right camera prim or path (default: /OpenXR/Cameras/right_camera)
- camera_properties: dict
  
  Dictionary containing the camera properties supported by the Omniverse kit to be set (default: {"focalLength": 10})

Get the recommended resolution of the display device
```
get_recommended_resolutions() -> tuple
```
Returns:
- tuple
  
  Tuple containing the recommended resolutions (width, height) of each device view. If the tuple length is 2, index 0 represents the left eye and index 1 represents the right eye

Set the pose of the origin of the reference system
```
set_reference_system_pose(position: Union[pxr.Gf.Vec3d, None] = None, rotation: Union[pxr.Gf.Vec3d, None] = None) -> None
```
Parameters:
- position: pxr.Gf.Vec3d or None, optional
  
  Cartesian position (in stage unit) (default: None)
- rotation: pxr.Gf.Vec3d or None, optional
  
  Rotation (in degress) on each axis (default: None)

Set the angle (in radians) of the rotation axes for stereoscopic view rectification
```
set_stereo_rectification(x: float = 0, y: float = 0, z: float = 0) -> None
```
Parameters:
- x: float, optional
  
  Angle (in radians) of the X-axis (default: 0)
- y: float, optional
  
  Angle (in radians) of the Y-axis (default: 0)
- x: float, optional
  
  Angle (in radians) of the Z-axis (default: 0)

Specify the meters per unit to be applied to transformations (default: 1.0)
```
set_meters_per_unit(meters_per_unit: float) -> None
```
Parameters:
- meters_per_unit: float
  
  Meters per unit. E.g.: 1 meter is 1.0, 1 centimeter is 0.01

Specify the transformations to be applied to the rendered images
```
set_frame_transformations(fit: bool = False, flip: Union[int, tuple, None] = None) -> None
```
Parameters:
- fit: bool, optionl
  
  Adjust each rendered image to the recommended resolution of the display device by cropping and scaling the image from its center (default: False). OpenCV resize method with INTER_LINEAR interpolation will be used to scale the image to the recommended resolution
- flip: int, tuple or None, optionl
  
  Flip each image around vertical (0), horizontal (1), or both axes (0,1) (default: None)

Teleport the prim specified by the given transformation (position and rotation)
```
teleport_prim(prim: pxr.Usd.Prim, position: pxr.Gf.Vec3d, rotation: pxr.Gf.Quatd, reference_position: Union[pxr.Gf.Vec3d, None] = None, reference_rotation: Union[pxr.Gf.Vec3d, None] = None) -> None
```
Parameters:
- prim: pxr.Usd.Prim
  
  Target prim
- position: pxr.Gf.Vec3d
  
  Cartesian position (in stage unit) used to transform the prim
- rotation: pxr.Gf.Quatd
  
  Rotation (as quaternion) used to transform the prim
- reference_position: pxr.Gf.Vec3d or None, optional
  
  Cartesian position (in stage unit) used as reference system (default: None)
- reference_rotation: pxr.Gf.Vec3d or None, optional
  
  Rotation (in degress) on each axis used as reference system (default: None)

Subscribe a callback function to the render event
```
subscribe_render_event(callback=None) -> None
```
The callback function (a callable object) should have only the following 3 parameters:
- num_views: int
  
  The number of views to render: mono (1), stereo (2)
- views: tuple of XrView structure
  
  A XrView structure contains the view pose and projection state necessary to render a image. The length of the tuple corresponds to the number of views (if the tuple length is 2, index 0 represents the left eye and index 1 represents the right eye)
- configuration_views: tuple of XrViewConfigurationView structure
  
  A XrViewConfigurationView structure specifies properties related to rendering of a view (e.g. the optimal width and height to be used when rendering the view). The length of the tuple corresponds to the number of views (if the tuple length is 2, index 0 represents the left eye and index 1 represents the right eye)
  
  The callback function must call the set_frames function to pass to the selected graphics API the image or images to be rendered
  
  If the callback is None, an internal callback will be used to render the views. This internal callback updates the pose of the cameras according to the specified reference system, gets the images from the previously configured viewports and invokes the set_frames function to render the views
Parameters:
- callback: callable object (3 parameters) or None, optional
  
  Callback invoked on each render event (default: None)

Pass to the selected graphics API the images to be rendered in the views
```
set_frames(configuration_views: list, left: numpy.ndarray, right: numpy.ndarray = None) -> bool
```
In the case of stereoscopic devices, the parameters left and right represent the left eye and right eye respectively. To pass an image to the graphic API of monoscopic devices only the parameter left should be used (the parameter right must be None)

This function will apply to each image the transformations defined by the set_frame_transformations function if they were specified

Parameters:
- configuration_views: tuple of XrViewConfigurationView structure
  
  A XrViewConfigurationView structure specifies properties related to rendering of a view (e.g. the optimal width and height to be used when rendering the view)
- left: numpy.ndarray
  
  RGB or RGBA image (numpy.uint8)
- right: numpy.ndarray or None
  
  RGB or RGBA image (numpy.uint8)
Returns:
- bool
  
  True if there is no error during the passing to the selected graphics API, otherwise False

Available enumerations

Form factors supported by OpenXR runtimes (XrFormFactor)
- XR_FORM_FACTOR_HEAD_MOUNTED_DISPLAY = 1
- XR_FORM_FACTOR_HANDHELD_DISPLAY = 2
Environment blend mode (XrEnvironmentBlendMode)
- XR_ENVIRONMENT_BLEND_MODE_OPAQUE = 1
- XR_ENVIRONMENT_BLEND_MODE_ADDITIVE = 2
- XR_ENVIRONMENT_BLEND_MODE_ALPHA_BLEND = 3
Primary view configuration type (XrViewConfigurationType)
- XR_VIEW_CONFIGURATION_TYPE_PRIMARY_MONO = 1
- XR_VIEW_CONFIGURATION_TYPE_PRIMARY_STEREO = 2
Reference spaces (XrReferenceSpaceType)
- XR_REFERENCE_SPACE_TYPE_VIEW = 1
- XR_REFERENCE_SPACE_TYPE_LOCAL = 2
- XR_REFERENCE_SPACE_TYPE_STAGE = 3
Action type (XrActionType)
- XR_ACTION_TYPE_BOOLEAN_INPUT = 1
- XR_ACTION_TYPE_FLOAT_INPUT = 2
- XR_ACTION_TYPE_VECTOR2F_INPUT = 3
- XR_ACTION_TYPE_POSE_INPUT = 4
- XR_ACTION_TYPE_VIBRATION_OUTPUT = 100

Available constants

Graphics API extension names
- XR_KHR_OPENGL_ENABLE_EXTENSION_NAME = "XR_KHR_opengl_enable"
- XR_KHR_OPENGL_ES_ENABLE_EXTENSION_NAME = "XR_KHR_opengl_es_enable"
- XR_KHR_VULKAN_ENABLE_EXTENSION_NAME = "XR_KHR_vulkan_enable"
- XR_KHR_D3D11_ENABLE_EXTENSION_NAME = "XR_KHR_D3D11_enable"
- XR_KHR_D3D12_ENABLE_EXTENSION_NAME = "XR_KHR_D3D12_enable"
Useful constants for applying haptic feedback
- XR_NO_DURATION = 0
- XR_INFINITE_DURATION = 0x7fffffffffffffff
- XR_MIN_HAPTIC_DURATION = -1
- XR_FREQUENCY_UNSPECIFIED = 0

feixuwu / semu.xr.openxr