Generalized API for external eye/lip/body-tracking devices
InsaneGrox opened this issue · comments
Currently the only way to get eye or lip tracking into neos without a bunch of hacky workarounds involving hacked drivers and or custom implementation that only works with a single avatar using websockets is by using an HTC Vive pro eye OR an HTC vive, HTC vive pro, or HTC vive cosmos with the Droolon F1 Eye tracker (which is only available in china outside of overpriced resellers selling for roughly $400 USD), and a currently unreleased lip tracking module, Opening up these data streams for external programs would make DIYing this type of thing far easier, and allow existing unsupported hardware (such as pimax's eye tracking, pico neo 2 eye, reverb G2 omnicept, ect) to be adapted by tinkerers to work inside of neos long before official support can be implemented.
@InsaneGrox- as was noted to you by @Frooxius in the Discord, That'd be a separate feature, but I could add some form of generalized API for external eye/lip tracking devices (or even body tracking), so you can just feed Neos the data.
This request could be potentially beneficial in adding additional hardware down the line, such as the aforementioned Reverb G2 Omnicept edition. Support for additional hardware sensors including the heart-rate monitors would also be beneficial in this capacity.
The best way to do this would be to define some simple protocol and then either have Neos connect to it on localhost over UDP/TCP or vice versa. That way you could setup a simple program and then just continually feed Neos the most recent data.
The main question is, what protocol would this be? Perhaps just JSON for simplicity? Maybe even use WebSocket, since that provides nicer way to work with text based messages out of the box. Would also Neos poll data or would they be pushed?
Could it be done via a plugin system instead that's easier for the end user than trying setting up a websocket to get it work which needs python and it's too complex but dragging in a dll is easy
It could when the plugins could be separated to user-side only (plugins that don't touch the datamodel) and plugins providing extra components and thus incompatible with unmodified clients.
We do have that on the roadmap here: https://github.com/Neos-Metaverse/NeosPublic/projects/12#card-61307662
Long term we'd like to add ability to load device driver plugins easily and open-source some of the existing integrations we have as samples. Don't have a timeline for this though.
Hey all, I've made a workaround for this: https://github.com/dfgHiatus/Neos-Eye-Face-API