Does not work but WebcamTexture does work

Question

Does not work but WebcamTexture does work

ARportals opened this issue 5 years ago · comments

Upreal Digital Inc. commented 5 years ago

I'm streaming with a Zcam E2 camera which has its own internal streaming converter software which outputs to a virtual camera. I'm able to see the feed with OBS AND with Unity's default WebcamTexture implementation (running low fps though at 9-20fps but still shows the image) but it does not work properly with AVProLiveCamera. The colors are off and it's running at 0.02 fps so there's definitely something wrong. If it's working with WebcamTexture, it should work with the AVProLiveCamera, right?

Your Setup (please complete the following information):

Unity version: 2019.2.0f1
AVPro Live Camera version: Version 2.8.0
Operating system version: Windows 10 64bit
Hardware/webcam model: ZCam E2
Video mode (resolution, frame-rate, etc): 1080p @ 30fps

To Reproduce
Steps to reproduce the behavior:

Install E2 Cam + Converter software (using Ethernet cable)
Open AVpro default scene
See error

Logs
Where can I find logs?

Andrew commented 5 years ago

Any luck?

Andrew · Answer 1 · Sat Feb 15 2020 01:01:43 GMT+0800 (China Standard Time)

Hi,

Thanks for reporting this problem.

We don't have a E2 camera to test with unfortunately...

I think the colors being wrong may be somehow caused by running Unity in Linear colorspace. Perhaps you could try changing this in Player settings to see whether it improves things? Or you could try one of the othe demos (like MaterialMapper) to see if that helps it, because in some of the demos we display the camera image using IMGUI which doesn't handle sRGB conversion correctly when running in Linear mode.

For the low frame rate...that is strange..Perhaps you could try setting the desired frame-rate value in the component? Or perhaps try in the CameraExplorerDemo scene to see whether you can set the frame-rate in there?

Thanks,

Deleted user · Answer 2 · Wed Feb 19 2020 11:19:19 GMT+0800 (China Standard Time)

Hi Andrew!

Thank you for replying and even following up! it's really appreciated!

So about the color being wrong, after testing in the MaterialMapping Demo scene, the issue is not there on those object so that's great because I was planning to use the feed on objects anyways! Not sure why I didn't realized this before!

The FPS is a little more tricky, I tried setting the desired fps in the AVPro Live Camera component but no luck there.

I also tried the Camera Explorer Demo like you suggested but I'm still getting a rediculusly low fps. (0.02fps aver)

I would have been surprise if you guys had a Zcamera haha! Is there any log or something similar that could help you figure it out?

What could be the cause of WebcamTexture working but not your plugin?

Thanks!

Andrew · Answer 3 · Sat Feb 22 2020 00:26:30 GMT+0800 (China Standard Time)

Thanks for the feedback! Yeah I'm not sure why the frame rate is so low in our plugin but not in WebcamTexture....We would love to fix that but i'm not sure how...could you share a screenshot of the CameraExplorerDemo scene running so I can see which modes / frames rates it enumerates?

Thanks,

Deleted user · Answer 4 · Sun Mar 01 2020 07:03:42 GMT+0800 (China Standard Time)

Hey Andrew!

Here's the screenshot:

Anything else you need that would help you?

Cheers,

Andrew · Answer 5 · Mon Mar 02 2020 20:59:45 GMT+0800 (China Standard Time)

Thanks for that.
Could you download GraphStudioNext:
https://github.com/cplussharp/graph-studio-next
(download links on that page in the readme section)

then go to Graph > Insert Video Source > Your Camera Name
Then right-click on one of the output "pins" eg:

go to Properties and see what options it has for color formats/ frame rates etc - maybe use alt-printscreen key to capture some screenshots of these for us to look at.

Thanks,

Andrew · Answer 6 · Mon Mar 02 2020 21:01:08 GMT+0800 (China Standard Time)

Also I can't really tell anything about the colours being wrong from your photograph. Perhaps you could use alt-printscreen keyboard shortcut to take a screenshot, and also send a screenshot of what it should look like. You can also click on the image preview in the CameraExplorerDemo to make it fullscreen, which will help too. At this point I suspect that the red and blue channels are swapped for some reason....Thanks,

Andrew · Answer 7 · Wed Mar 11 2020 00:03:26 GMT+0800 (China Standard Time)

Hi @arthamas any further information?

Deleted user · Answer 8 · Fri Mar 13 2020 23:17:29 GMT+0800 (China Standard Time)

Hey Andrew, I haven't forget about that issue! I'll send you what you requested tonight!

Deleted user · Answer 9 · Sun Mar 15 2020 05:33:23 GMT+0800 (China Standard Time)

Hey Andrew,

Thanks for the follow-up!

Here's the requested information :

Deleted user · Answer 10 · Sun Mar 15 2020 05:45:41 GMT+0800 (China Standard Time)

Oh and here's the image was taken using the Explorer Demo after clicking on the feed. The colors of the tools in the image are the right color so I don't think the colors are simply inverted. But we can see on the dark zone outside of the lens that there are some color artifacts, it seems like there's not enough color information or something. Not sure but it easier to see with this last screenshot.

Deleted user · Answer 11 · Thu Apr 02 2020 02:21:33 GMT+0800 (China Standard Time)

Hey @AndrewRH , any news on this?

Deleted user · Answer 12 · Fri May 29 2020 02:46:45 GMT+0800 (China Standard Time)

Following up again! I would really appreciate some fix on that!

Andrew · Answer 13 · Fri May 29 2020 03:31:09 GMT+0800 (China Standard Time)

It looks like an issue caused by using the Linear color space. Perhaps you could go into Player Settings > Other and change the color space to Gamma.

Does that help?

Andrew · Answer 14 · Sat May 30 2020 01:28:58 GMT+0800 (China Standard Time)

Hi,

I think I know what the problem is. One of our shaders aren't applying gamma correction.

I've attached the fixed shader here. Please override your existing file with this one and let know if it solves the issue.

Thanks,
AVProLiveCamera_YUV_YV12.zip

Deleted user · Answer 15 · Sat May 30 2020 02:01:59 GMT+0800 (China Standard Time)

Hi Andrew,

Thank you for following up! Ok I'll give your modified shader a try!

Would this help the low frame rate too?

Cheers,

Deleted user · Answer 16 · Sat May 30 2020 02:40:34 GMT+0800 (China Standard Time)

@AndrewRH Ok so just did a test with the new shader and the colors are great now! Great work :)

Now my only issue is the low frame rate. It seems like it can't find the FrameRate and the Format is empty like you can see in the following image (The 30fps is not green and there's no format)

Deleted user · Answer 17 · Sat May 30 2020 02:44:58 GMT+0800 (China Standard Time)

Here's an image of the streaming converter app that generates decodes the image stream. This does work with regular webcam Texture but it's too heavy. I tried other resolution/fps and I get the same result. (4K Cinema or regular 4K etc..)

Deleted user · Answer 18 · Sat May 30 2020 02:51:13 GMT+0800 (China Standard Time)

I get like 1 frame every couple of seconds

Andrew · Answer 19 · Sat May 30 2020 03:25:49 GMT+0800 (China Standard Time)

That's great! I'm glad the colors are correct now.

For the frame rate - could you check Task Manager > Performance and see if there is anything that is maxing out - eg CPU or GPU?

What specs are you running?

Thanks,

Andrew · Answer 20 · Mon Jun 01 2020 03:11:36 GMT+0800 (China Standard Time)

@arthamas as well as that, could you also try upgrading to the latest vresion of the Z Cam Stream Converter software (version 0.58 is the latest I believe from their website) and see whether that improves the frame rate?

http://www.z-cam.com/software-update/

Thanks,

Deleted user · Answer 21 · Mon Jun 01 2020 03:47:38 GMT+0800 (China Standard Time)

@AndrewRH I actually did that already last night and I can't even get an image now with either Webcam Texture or AVProLiveCamera. I'm just getting a grey image at 30 fps even though it's working in OBS. In the last version, I had that issue but after adding a dummy "DevicePath"="" in windows registry of the virtual cams, it fixed it. But in the new version, it does not seems to work... Not sure why it would work in obs but not in Unity. If you have any insight it would be amazing. They added a super large resolution of 3696 x 2772 that I would LOVE to use! So far in OBS it's already dropping frame at that resolution so not sure how powerful my CPU needs to be in order to have smooth playout.

Thanks a lot!

Andrew · Answer 22 · Mon Jun 01 2020 04:36:03 GMT+0800 (China Standard Time)

Oh no...
I really wish I had that device so I could test it directly with the code.
In that case, perhaps you can go back to the previous version of Stream Converter, and I could send you an update to the plugin to test which could resolve the frame-rate issue?

Thanks,

Deleted user · Answer 23 · Tue Jun 02 2020 04:40:08 GMT+0800 (China Standard Time)

@AndrewRH Yes, that would definitely be ideal if you had the physical! Are you guys in the US? Maybe I could get one camera to your offices...
Ah yes, I think that's a great plan! I would definitely test that update of the plugin whenever you send it to me.

Andrew · Answer 24 · Tue Jun 02 2020 07:56:26 GMT+0800 (China Standard Time)

Ahhh US - no, we're in Scotland UK.
I am working on an update to the plugin. I will post it here for you to try when it's ready soon.

Thanks,

Deleted user · Answer 25 · Wed Jun 03 2020 07:04:39 GMT+0800 (China Standard Time)

Oh in Scotland! Maybe a little far!
Ok great! Looking forward to it!

Andrew · Answer 26 · Thu Jun 04 2020 23:36:40 GMT+0800 (China Standard Time)

Hi @arthamas
I have uploaded a new version here for you to try with your ZCAM. It's version 2.8.2. Please make sure that the correct version logs in the Console Window after you have updated. You may need to update from a freshly opened Unity so that DLL files can be updated.

UnityPlugin-AVProLiveCamera-v2.8.2-Trial.zip

I suspect that the way we did frame rate selection may have been causing the low frame rate issues you experienced, so I've changed how all of that works. Hopefully it will behave better.

When you run the demo scenes it will no longer try to force the highest frame rate possible, but will instead just use the default frame rate of the device.

You can now select the frame rate manually. In the Camera Explorer demo, once you have selected your mode, a Frame Rate menu will appear:

In the other demos that use the Quick Device Menu, there is a new column where you can change frame rate:

Do let me know how that goes.

Thanks,

Deleted user · Answer 27 · Fri Jun 05 2020 04:39:53 GMT+0800 (China Standard Time)

Hey @AndrewRH !

Thanks for the new version!

I just gave it a try and I get similar results as before sadly. I tried with both versions of ZCam Stream Converter and the framerate is still pretty low. I installed your unity package with the update on a fresh Unity Project to avoid dll issues. I took a video of my test that you can download using this wetransfer.com link: https://we.tl/t-OYIRxBRQ0s

I do still think your right that the frame rate selection is at cause because I can't even select the 30fps in the menu (it remains grey)

So with the 0.46 version, I still get a low framerate and with 0.58 version I still get grey with a little tiny text that moves in the screen.

I'm wondering... just had a thought... do you know if there's a way for me to stream my feed from my Zcam over to web to you so that you can debug like that? That would be simpler than guessing!

I can stream using RTMP with camera maybe that would help?

Cheers

Andrew · Answer 28 · Fri Jun 05 2020 05:15:41 GMT+0800 (China Standard Time)

Thanks I appreciate you testing that and showing me via video - it's very clear. I'm afraid I don't think that setting up a stream would help us work on the issue.

I really thought it was going to work...

I have another idea to test. I want to see whether basic DirectShow can handle it. I asked you use download and test GraphStudioNext software before. If you could open it again, and add your camera as the source. The right-click on the "Video" output pin and select "Render" then press the Play button and see what sort of frame-rate you get.

Also - have you seen it working at full-frame rate in any other apps before? And when running in Unity does Task Manager show and bottlenecks like large CPU / GPU usage?

Thanks,

Deleted user · Answer 29 · Fri Jun 05 2020 22:58:55 GMT+0800 (China Standard Time)

@AndrewRH Thank you again for your update though! It was a nice try!

So I tried the test you suggested and it gave me the following graph:

When I hit play I only get 1 image and that's it. I also tried the same thing with the OBS-Virtual camera that always work in Unity and I can't even get an image with that one so maybe I'm missing something here... I also tried the 32 bit version of the GraphStudioNext just to see and I get the same result except for the fact tha I can only see the OBS virtual camera and not any of the ZCam virtual camera in the 32 bit version. I was expecting some kind of H264 Decompressor node instead of the AVI Decompressor but I guess the codec is part of the AVI Decompressor?

To answer your other question, I have seen it working at pretty near full-frame rate in OBS and when I use Unity Webcam Texture it's dropping frame but working. Let me do a quick test right now with Webcam Texture within Unity and share that with you.

Deleted user · Answer 30 · Fri Jun 05 2020 23:16:41 GMT+0800 (China Standard Time)

@AndrewRH Just did a quick test with the WebcamTexture inside Unity that includes Task Manager performance in the video:

https://we.tl/t-BioOgbsa6S

I added couple of checks recently and just found out that Unity Webcam Textures does not find any available resolution on the Z Cam Virtual Camera... Maybe a similar fix with the Resolution that you did with the Framerate by using the default one would help?

Cheers,

Deleted user · Answer 31 · Sun Jun 21 2020 08:16:02 GMT+0800 (China Standard Time)

@AndrewRH Just following-up!

Andrew · Answer 32 · Mon Jun 22 2020 00:50:46 GMT+0800 (China Standard Time)

Oh wow, sorry I somehow didn't see your updates on this thread! Apologies

Thanks for checking that - let me get my thoughts together and I will get back to you.

Andrew · Answer 33 · Mon Jun 22 2020 00:51:24 GMT+0800 (China Standard Time)

Could you possibly send the video file again? The link has expired...

Andrew · Answer 34 · Tue Jun 23 2020 05:28:05 GMT+0800 (China Standard Time)

Hi @ARportals

I wonder if we could arrange a remote access session using something like TeamViewer?

I think if I could have access to a machine running your camera, and using the GraphStudioNext (and perhaps Unity) then I may be able to work out what we need to change to better support the camera.

I think it would only require 15-30 minutes of access.

If this is something you're happy to try then please let us know a date and time that is suitable for you. If you contact our product support email address listed here and reference this issue then they can forward my email address to you to set up the session.

Thanks,

Deleted user · Answer 35 · Sun Jun 28 2020 07:01:17 GMT+0800 (China Standard Time)

Hi @AndrewRH ,

No problem! I sadly don't have the video file any more of my test.

Yes off course! A remote session with Teamviewer does work for me. Sorry for the late reply, I have been quite busy!

I just sent an email for the remote Teamviewer session. I'll make sure you'll have access to Unity and the camera etc during the remote session. You can definitely take more than 15-30 minutes in case you need it for sure.

Cheers,

Andrew · Answer 36 · Tue Jun 30 2020 21:09:42 GMT+0800 (China Standard Time)

Thanks @ARportals

I tried to email you bit it just bounces back saying

The response from the remote server was:
550 No Such User Here

I will try again later

Deleted user · Answer 37 · Wed Jul 01 2020 05:46:17 GMT+0800 (China Standard Time)

@AndrewRH Just fixed my email issue. It should be good now.

Andrew · Answer 38 · Tue Jul 07 2020 05:27:13 GMT+0800 (China Standard Time)

Thanks we finally resolved this issue. It seemed to be related to some weird handling of the SyncClock in the DirectShow graph. Disabling the clock made it run at the correct speed. We will release a new version of the plugin with the option to disable the clock. Thanks for all the help @ARportals I will close this issue now and we will release the fixed version soon. If you have any other issues just let us know.

Thanks,