Question: mixed reality capture
mluketin opened this issue · comments
Hi,
Can you please provide an example on how mixed reality capture can be achieved using winrtc? In m71 sdk, VideoCapturerCreationParameters had EnableMrc property, but I don't see anything like that in the patches for winrtc (I see the media_capture_video_profile and record_media_description that were added to VideoCaptureCapability in patch 3004, but nothing regarding mixed reality capture).
Thank you!
According to the information provided here in #36 mixed-reality capture is only possible with MixedReality-WebRTC project, and far as I understand the information it would be the same no matter which webrtc implementation would be used under the hood - at the moment they use patched m71 version, target is to use winrtc version based on m84. The problem is that all the developers in both repositories keep silent, so I'm not even sure that the advertised switch to winrtc would happen in MixedReality project.
As a side note - at least this implementation works fine with camera capture on Hololens 2, if you're more ready to C++ than C#.
@SeraphicGreevil Hello, may I ask you a question? Can I build winrtc into an ARM64 DLL for unity to call, and finally deploy it to Hololens 2? I used MixedReality-WebRTC before, but unfortunately, for Hololens 2, it only supports ARM build about MixedReality-WebRTC. Hololens2's Research Mode only supports ARM64 build, so I can't use MixedReality-WebRTC and Research Mode in one project at the same time (see #22). If winrtc can be built into ARM64 DLL, I can use WebRTC and Research Mode in unity project at the same time.
plus, I only want to get camera stream on Hololens 2, don't need mixed reality stream.
Any help will be very appreciated!
Thanks in advance.
@ylsislove well not sure where to start, since I don't know what libraries are used in unity code behind webrtc implementation. All in all, yes you can build arm64 version and it would function on Hololens, but it is C++ project. As far as I understand, at the moment you cannot just replace webrtc.lib in MixedReality-WebRTC solution with this one and hope that it automagically works, since that implementation relies on additional code specific to Hololens. See, I was also targeting Hololens 2, but my intention was to build libmediasoupclient library, which is also pure C++ and later on craft a C++/WinRT video-call client using mediasoup as a higher-level wrapper above webrtc code. In theory you could write your own C# library wrapping up winrtc code using P/Invoke, but technically this would mean that you'll need to implement some subset of MixedReality-WebRTC C# code by your own. In any case you could give it a try, just keep in mind, that it's not an easy task - API porting is rather dull and monotonous job. As a side note, most probably you'll also ought to write some C++ code above winrtc, because webrtc requires 3 native threads to function properly and I doubt you'll be able to create these using P/Invoke, but here I'm not 100% sure.
@SeraphicGreevil Thanks for the quick reply! I'll look into it asap.
@ylsislove well not sure where to start, since I don't know what libraries are used in unity code behind webrtc implementation. All in all, yes you can build arm64 version and it would function on Hololens, but it is C++ project. As far as I understand, at the moment you cannot just replace webrtc.lib in MixedReality-WebRTC solution with this one and hope that it automagically works, since that implementation relies on additional code specific to Hololens. See, I was also targeting Hololens 2, but my intention was to build libmediasoupclient library, which is also pure C++ and later on craft a C++/WinRT video-call client using mediasoup as a higher-level wrapper above webrtc code. In theory you could write your own C# library wrapping up winrtc code using P/Invoke, but technically this would mean that you'll need to implement some subset of MixedReality-WebRTC C# code by your own. In any case you could give it a try, just keep in mind, that it's not an easy task - API porting is rather dull and monotonous job. As a side note, most probably you'll also ought to write some C++ code above winrtc, because webrtc requires 3 native threads to function properly and I doubt you'll be able to create these using P/Invoke, but here I'm not 100% sure.
Hello, can I ask your some question? could you please tell me in detail how you realized the audio and video communication between HoloLens and other computer applications? Thank you very much!