'immersive-vr' blend mode value on additive light displays

[Nell Waliczek]

It seems like this topic is less resolved than it had originally seemed, keeps getting tacked on to other issues/pull requests, and is unfortunately quite contentious. Let's move the discussion here to avoid polluting other discussions or having important comments get lost on pull requests that have already been merged.

The problem at hand is specifically regarding additive light display support for 'immersive-vr':

• reporting 'opaque' isn't factually accurate
• reporting 'additive' may encoure developers not to request 'immersive-vr'

Gathering the comments/concerns from various discussions:

1. Some user agents will choose to deny 'immersive-vr' sessions on additive light displays
2. Reporting a false value ('opaque') means developers will be unable to adjust their rendering to be tailored to the display characteristics
3. No device is planning to support 'immerisve-vr' before they are able to support 'immersive-ar'
4. Decisions must be left to the user agent to decide the correct path so long as lack of conformance doesn't create an undue burden on developers or content paths that function incorrectly on a subset of hardware
5. A developer requesting 'immersive-vr' will not find out the blend mode until after the session has been created. Specifically, this means the user will almost certainly have gotten a consent prompt, any other 'immersive' experience will have been kicked out of being visible on the primary display, and potentially other consequences due to the "immersiveness" of immersive sessions.
6. It is unclear if 'immersive-vr' sessions will be allowed to enable real-world-understanding features. If this does happen, developers may choose to do so instead of requesting an 'immersive-ar' session.
7. One consequence of a developer "faking" an AR session using 'immersive-vr' would be unexpected or incorrect behavior on a display which does not support 'immersive-ar'. This could be as egregious as as spinning up a VR session on a VR device with an external display only to immediately terminate the session because the blend mode doesn't match.
8. Another consequence of a developer "faking' an AR session using 'immersive-vr' is that devices which support AR through alpha blending will not know that is the developers intent. This will result in a similar problem to the one described in bullet 7. (And is related to, but not the same as, #15)

Given the contentious history of this issue, I'd like to remind everyone to take the time to consider other people's points of view before responding quickly. I would also like to STRONGLY suggest that agreement be reached on the importance each of the "concern" bullets before things digress into technical solutions.

[Nell Waliczek]

Copying content from @thetuvix's comment on #15 (comment)

There's an additional question as to what an "immersive-vr" session returns on an additive-light display:

* Do we return `"opaque"` (or perhaps even null) since your app is not _intending_ to do environment blending if you chose `"immersive-vr"`?

* Do we return `"additive"` because that is what the device will actually do? It cannot opaquely block out the real world, and so the app may wish to know that it needs to adjust its logically-VR content accordingly, e.g. lighten dark objects that would otherwise be near-invisible.

My vote is to return "additive" for such an "immersive-vr" session.

Consider a logically VR app, such as a 360-degree video app or HoloTour. This app should not select "immersive-ar", because it definitely does not want to enable video composition of the user's environment if on a device that can truly stay opaque. However, that app still may want to adapt its content or UI for the display that's in use (e.g. change its UI controls from black to 50% gray), and so the distinction is still meaningful.

Especially if Magic Leap does disable "immersive-vr" content, even just for a few months, that should help guide truly AR sites to use "immersive-ar" as intended without requiring us to lie or omit information in "immersive-vr".

[Nell Waliczek]

Copying content from @rcabanier's comment on PR #12 (comment)

We don't right now but if a good portion of AR content takes that other path, we would eventually be forced to support it (unless we take over the entire AR market :-P)

What would lead you to believe that a good portion of AR content WOULD take that other path

Authors will take the easiest way. If the AR device they own returns additive as a blend mode, they will use that and ignor asking for 'immersive-ar'.
@toji 's explainer had this as an example so they might think it's ok to do.

(To reiterate: my problem with keying of the blendmode attribute is that this allows 2 different workflows to create AR content)

("asking for immersive-vr and being fine with a non-opaque blend mode" is the other path you're referring to, right?) ?

yes

Such a path would deliberately put them in a non-AR mode in a handheld device, for example. Would you be more comfortable having this be an option that has to be explicit?

Yes. If there was a way for an author to say "I know what I'm doing and will request AR AND VR" in 1 call that would be great.
That does bring up the question what a device that can do both would do. Would we let that decision be UA dependent?

I do think that you're disregarding the case where a developer just wants to present a model, and is less worried about whether that's AR in a see-thru headset or VR in a traditional VR headset; for example, if I write an app to render an exploded model of an engine, I'd be fine with it being presented in see-thru AR if that's how you can present it, or I'd be fine with it in VR if that's the mode available, too. We shouldn't make this impossible, or even hard; we SHOULD make it clear to the developer what they're asking for, since that would change what a hand-held device would do.

The current API allows that; the author just has to ask for AR first and if not supported, then for VR. (I'm assuming that by "I" in this paragraph, you mean "the author" and not "the user".)

It's a bit clunky but a reasonable decision.

Although I understand the desire from your perspective to not simply support immersive-vr, I do expect some other see-thru headset vendor might believe that supporting immersive-vr is the right choice for them to make.

As long as they don't return additive for blend mode, that would be a reasonable choice.

As for handheld devices, is Chrome going to allow immersive-vr even on devices that don't support VR?

[Brandon Jones]

Comment moved from #12 at @NellWaliczek's request:

As for handheld devices, is Chrome going to allow immersive-vr even on devices that don't support VR?

The answer depends on what you mean by "devices that don't support VR"? While we will support Daydream on devices that are Daydream Ready, our aim is to support immersive-vr on as many other mobile devices as possible via Cardboard mode.

We have no plans at this time to have a "fullscreen VR" mode (ie: mobile AR style presentation without the passthrough feed.) Nor do we intend to support immersive-vr on desktops/laptops that have no VR hardware attached.

[rcabanier]

@NellWaliczek I agree with your points (especially 6, 7, 8)
My vote is for not reporting a blend mode at all if the user asked for an immersive VR session.

Here is why I think having a blendmode for immersive-vr is bad:

1. It provides 2 ways of creating AR content and confusion if certain AR features should be available for VR (point 6)
2. If a UA decides to just implement the WebXR spec (because they don't care about AR), they will be forced to implement blendmode because authors will expect that the attribute exists.
3. It makes the AR spec simpler. We can simply drop the opaque blend mode and don't have to redefine VR behavior.
4. It creates a workflow where the UA warns the user that the device is not VR when immersive-vr is requested but the page chooses the correct path anyway by checking blend mode. This might confuse the user and make them ignore the warning.
5. It creates confusion for passthrough devices such as the Vive Pro or Varjo XR-1. If a lot of content starts to switch on the blend mode, should they turn the cameras on even when VR is requested? (see [this comment](#12 (comment) for more info)
6. Some UAs don't want to allow immersive-vr sessions on their AR device because most VR content looks bad since most authors won't make an effort to make it AR friendly. However if enough AR content chooses the VR path with a switch on blendmode, those UAs will then be forced to support that workflow (and get user complaints that WebXR looks bad)
7. If an author is savvy and enforces that they don't want their VR experience to work in an AR devices, they could abort the session when they see the additive blendmode. This will appear as a broken experience to the user.
8. UAs don't want to lie by reporting opaque if their device is really additive

Maybe we can discuss this during TPAC and have a vote :-)

[Alex Turner]

Regarding the concerns @NellWaliczek listed above:

6. It is unclear if 'immersive-vr' sessions will be allowed to enable real-world-understanding features. If this does happen, developers may choose to do so instead of requesting an 'immersive-ar' session.

One key meaning of immersive-vr vs. immersive-ar that we've discussed is whether passthrough video composition should be enabled on hardware that supports it. That seems strictly orthogonal to whether the app should be able to access world mesh, etc. for placing content. For example, just because an app is using opaque environment blending does not mean the app's only world reasoning should be a local-floor bounds rectangle: for example, an app that can access world planes/mesh could generate its virtual world based on your environment.

3. No device is planning to support 'immerisve-vr' before they are able to support 'immersive-ar'

Is this true about video-passthrough headsets like the Varjo XR-1? The desktop browsers that target OpenVR and thus can support this device today would support immersive-vr. However, over time as those browsers add support for video-passthrough headsets, the headset would gain support for immersive-ar sessions.

Regarding @rcabanier's responses:

4. It creates a workflow where the UA warns the user that the device is not VR when immersive-vr is requested but the page chooses the correct path anyway by checking blend mode. This might confuse the user and make them ignore the warning.

It seems to me that such a warning would reasonably guide developers to the right path. Their AR content could technically show up on the AR device they've got when they incorrectly specify "immersive-vr", but their users will see annoying warnings. If they simply change "-vr" to "-ar", the experience will be seamless like the other AR WebXR content they see on that device.

This may illuminate a key pivot that affects the design space for solutions here:

• If a minimal AR module will land in time to intercept WebXR public availability on AR headsets, we can ensure that "immersive-ar" is clearly a superior experience for users, so that developers naturally choose the right path, which is even easier to do than their blend mode hack. (e.g. some AR headsets block "immersive-vr" content - other AR headsets show warnings)
• If a minimal AR module will not land in time to intercept WebXR public availability on AR headset, we've already prevented some of the problems described here by removing environmentBlendMode from the core spec. This means that developers are equally blocked from requesting an "immersive-ar" session as they are from hacking in an "additive" check. If browsers introduce "immersive-ar" and "additive" at the same time, we may avoid many of the concerns here.

[Manish Goregaokar]

The conclusion at TPAC was to have environmentBlendMode accurately represent the actual blending happening regardless of the mode.

The current spec text says

This technique MUST be applied on [=additive light=] displays, regardless of the [=XRSession/mode=].

which matches our conclusion, so I'm going to close this issue.

Thanks, all, for bearing through this long discussion! 😄