What is the environment blend mode for devices that optionally support camera passthrough?

[Manish Goregaokar]

See preexisting discussion at #12 (comment)

What does a device that optionally supports video passthrough (like a Vive Pro or Varjo XR-1) return for XRSession.environmentBlendMode?

1. Do we return "opaque" if you created an "immersive-vr" session (which does not turn on the passthrough camera) and "alpha-blend" if you created an "immersive-ar" session (which does turn on the passthrough camera)?
2. Do we return "alpha-blend" even for "immersive-vr" sessions, because you could have turned on video passthrough by creating an "immersive-ar" session?
3. Do we return "opaque" even for "immersive-ar" sessions, because it is the "natural" state of the device?

Once a developer is already in an XRSession, I believe they will care most about how that session will actually behave. My suggestion is that XRSession.environmentBlendMode be defined to represent the "effective environment blend mode" for the active session, informed both by the XR device that's in use and the XRSessionMode that the app selected.

This is halfway stated in the bullets above:

The {{XRSession/environmentBlendMode}} MUST NOT be {{opaque}} for {{XRSessionMode/"immersive-ar"}} sessions.

When an [=xr device=] [=performs automatic composition=] using a [=passthrough camera=] in an {{XRSessionMode/"immersive-ar"}} session, it MUST report {{XREnvironmentBlendMode/"alpha-blend"}} blending behavior.

We may need to explicitly state the opposite in the "opaque" bullet above, such as:

When an [=xr device=] with a [=passthrough camera=] that's in an {{XRSessionMode/"immersive-ar"}} session is not [=performs automatic composition|performing automatic composition=], it MUST report {{XREnvironmentBlendMode/"opaque"}} blending behavior.

And then here, we may want to be more explicit that the environmentBlendMode is only "alpha-blend" when in such an "immersive-ar" session:

Such a device MUST have an {{XRSession/environmentBlendMode}} of {{alpha-blend}} if and only if that session is an {{XRSessionMode/"immersive-ar"}} session.

Originally posted by @thetuvix in #12

[Alex Turner]

There's an additional question as to what an "immersive-vr" session returns on an additive-light display:

• Do we return "opaque" (or perhaps even null) since your app is not intending to do environment blending if you chose "immersive-vr"?
• Do we return "additive" because that is what the device will actually do? It cannot opaquely block out the real world, and so the app may wish to know that it needs to adjust its logically-VR content accordingly, e.g. lighten dark objects that would otherwise be near-invisible.

My vote is to return "additive" for such an "immersive-vr" session.

Consider a logically VR app, such as a 360-degree video app or HoloTour. This app should not select "immersive-ar", because it definitely does not want to enable video composition of the user's environment if on a device that can truly stay opaque. However, that app still may want to adapt its content or UI for the display that's in use (e.g. change its UI controls from black to 50% gray), and so the distinction is still meaningful.

Especially if Magic Leap does disable "immersive-vr" content, even just for a few months, that should help guide truly AR sites to use "immersive-ar" as intended without requiring us to lie or omit information in "immersive-vr".

[Nell Waliczek]

@thetuvix and @Manishearth, based on prior experience with the Immersive Web group, you will likely find the discussions will go more smoothly if we limit each GitHub issue to a single topic. The title of this issue is specifically regarding pass-through devices. Please consider moving the discussion of the behavior of 'immersive-vr' on additive light displays to its own issue.

[Nell Waliczek]

I may have jumped the gun here, but went ahead and filed a dedicated issue to discuss the second topic raised by @thetuvix

[rcabanier]

@Manishearth with the current spec wording I believe it is clear what such a device should return.
If the cameras is on and it blends using source-over with the real world, it should return "alpha-blend". If it doesn't blend, it should be "opaque"

[Manish Goregaokar]

@thetuvix do you agree with @rcabanier's comment?

if so, perhaps we can close this? Alternatively, we can add a note to the spec that clarifies this

[Brandon Jones]

I completely agree with @rcabanier's comment. The thing that matters most to the developer is the current session behavior and not what capabilities it might have under a different configuration.

[Manish Goregaokar]

(This is my position too, if that wasn't clear. I'm okay with adding a note if people are still finding this ambiguous, but I feel that the current behavior is basically the "effective EBM" that Alex talks of)

[Brandon Jones]

I think the current text does indeed describe the behavior that we all seem to agree on here, but I put up a really minor (two word) PR at #26 that should make it completely unambiguous and hopefully close out this issue.