Flag to indicate the pose has been mostly emulated.
jespertheend opened this issue · comments
Devices like the valve index controllers have capabilities to emulate hand joints, but the generated skeleton is not detailed enough to perform any advanced gestures.
For accurately tracked hands for instance it is possible to precisely touch the tips of two fingers together. This isn't always possible with the generated skeletons by the valve index or oculus touch controllers for instance.
Should there be an exposed flag somewhere to let applications know that the skeleton should not be used for any accurate gestures, but for mostly rendering only?
So I'll firstly note that we have the emulatedPose flag on XRPose in the core spec that will be somewhat useful here.
But I don't actually think hand models that cannot precisely touch fingers are incapable of doing gestures, they're just hard to do that specific gesture with. This isn't a simple boolean, there's a confidence level here.
I believe the older OpenXR API had something about this but it was something you'd request on initialization, and it was removed later (cc @fordacious)
Devices like the valve index controllers have capabilities to emulate hand joints, but the generated skeleton is not detailed enough to perform any advanced gestures.
For accurately tracked hands for instance it is possible to precisely touch the tips of two fingers together. This isn't always possible with the generated skeletons by the valve index or oculus touch controllers for instance.
When have you seen this? Touching the fingers of the same hand should definitely work.
Should there be an exposed flag somewhere to let applications know that the skeleton should not be used for any accurate gestures, but for mostly rendering only?
I think it's the opposite: WebXR Hands is mostly useful for detecting gestures. The rendering must not exactly match the user for privacy reasons.
When have you seen this?
To be honest, the index was just an example. I’m not actually sure about the possibilities with index controller apis. But if you take the oculus touch controllers for instance. I believe there are only sensors for whether the user is touching a button, trigger or thumbstick. So if the user is able to do an ‘ok’ gesture is up to the shape of the controller and emulated hand shape. And even if a certain gesture could be guaranteed, the user probably expects different controls when they are using a physical controller vs optical hand tracking. Grabbing objects with a physical button vs making a grab pose with your hand for instance.
WebXR Hands is mostly useful for detecting gestures.
I agree, perhaps controllers like the index and oculus touch are not a good fit for this proposal, and shouldn’t expose any hand shapes at all.
But since it’s common to emulate hand poses for certain inputs for many VR experiences, I figured it could be nice if the ua did this for us. There are many different types of controllers each having different input types, and many more will probably be made in the future. And since the current WebXR spec only exposes a few buttons for controllers, it would be incredibly difficult for applications to emulate hand poses with limited information.
I guess what I’m asking is; should there be a way for applications to distinguish optical hand tracking from one where the user is still holding controllers? Since both types require a different approach in how the application handles input.
I guess what I’m asking is; should there be a way for applications to distinguish optical hand tracking from one where the user is still holding controllers? Since both types require a different approach in how the application handles input.
Oculus definitely doesn't have the ability to track hands while they are holding the controllers. Do you think there's a strong use case for this? Also, are you aware of any systems that support it?
When have you seen this?
To be honest, the index was just an example. I’m not actually sure about the possibilities with index controller apis. But if you take the oculus touch controllers for instance. I believe there are only sensors for whether the user is touching a button, trigger or thumbstick. So if the user is able to do an ‘ok’ gesture is up to the shape of the controller and emulated hand shape. And even if a certain gesture could be guaranteed, the user probably expects different controls when they are using a physical controller vs optical hand tracking. Grabbing objects with a physical button vs making a grab pose with your hand for instance.
When there is a hand object, there is no regular controller. I believe we report it as "oculus hand" instead of "oculus touch".
The expectation is that sites that support both, have different experiences depending if the user picks up a controller or uses their hands.
I'm unsure how this works for controller-less UAs like the Hololens...
WebXR Hands is mostly useful for detecting gestures.
I agree, perhaps controllers like the index and oculus touch are not a good fit for this proposal, and shouldn’t expose any hand shapes at all.
But since it’s common to emulate hand poses for certain inputs for many VR experiences, I figured it could be nice if the ua did this for us. There are many different types of controllers each having different input types, and many more will probably be made in the future. And since the current WebXR spec only exposes a few buttons for controllers, it would be incredibly difficult for applications to emulate hand poses with limited information.
That sounds reasonable. Can you file a separate issue to have recognized hand poses?
Maybe we can address those in the next version of the spec.
Oculus definitely doesn't have the ability to track hands while they are holding the controllers. Do you think there's a strong use case for this? Also, are you aware of any systems that support it?
The kind of behaviour I'm talking about can be seen in the home screen on the quest (unless the virtual environment setting has been set to pass-through) Here's an example:
com.oculus.vrshell-20201227-152603.mp4
A use case for this could be rendering the emulated hands in a game for extra immersion while still relying on button presses for input. Native VR applications already do this (some examples are Half Life Alyx and to a certain degree Shadow Point) but they have the advantage of knowledge about their exact make and model of controllers. So they can have carefully crafted poses based on the shape of the controller and available sensors.
WebXR inputs don't always have this advantage. I suppose it could be possible to make a library of poses for different controllers using the profiles attribute on input sources, but this requires a lot of work for developers and isn't forward compatible. I think it makes more sense to make use of the same system that is already provided with the hand input spec.
Can you file a separate issue to have recognized hand poses?
I'm not really sure I understand. What should the separate issue be about?
Oculus definitely doesn't have the ability to track hands while they are holding the controllers. Do you think there's a strong use case for this? Also, are you aware of any systems that support it?
The kind of behaviour I'm talking about can be seen in the home screen on the quest (unless the virtual environment setting has been set to pass-through) Here's an example:
Ah. That's not really derived from the tracking cameras. Those hand motions are triggered by touching the controller.
I'm unsure if touching the buttons is surfaced in WebXR.
WebXR inputs don't always have this advantage. I suppose it could be possible to make a library of poses for different controllers using the profiles attribute on input sources, but this requires a lot of work for developers and isn't forward compatible. I think it makes more sense to make use of the same system that is already provided with the hand input spec.
If you're looking for touching the controllers, the hands spec is likely not the correct place.
Can you file a separate issue to have recognized hand poses?
I'm not really sure I understand. What should the separate issue be about?
The issue would be to detect hand gestures ie thumbs up, down, etc
@jespertheend applications can render hands, based on controllers state, just like Index Knuckles often do, or good example would be Half Life: Alyx. They are not using Optical Hand tracking, and WebXR Hand Tracking API will not provide joints information for them. Those are application level implementation of hand model, not based on joints data, but based on assumtion by artist of controller shape, the way it is handled and different finger states based on ux of the controller.
This basically has nothing to do with optical hand tracking, that this API is about.
This basically has nothing to do with optical hand tracking, that this API is about.
In that case I think a guarantee needs to be made in the spec that no joint data will be exposed when the user is using such a controller. It seems like #29 will be taking care of this though.
In that case I think a guarantee needs to be made in the spec that no joint data will be exposed when the user is using such a controller. It seems like #29 will be taking care of this though.
Thanks for the reminder. Let's discuss in #29 with @Manishearth if that limitation is needed.
@Manishearth can this issue be closed?