[Flutter] Push video frame does not work before StreamPublishState.pubStatePublished
locandpad opened this issue · comments
Describe the bug
When pushing a ExternalVideoFrame using pushVideoFrame, I noticed that onVideoPublishStateChanged only changed to pubStatePublishing, I had to trigger another pushVideoFrame call (same input) to force the publishing state to pubStatePublished for it to work correctly.
To Reproduce
Steps to reproduce the behavior:
- Use the provided example from this repo
- Select
PushVideoFramefrom Advanced category - Join a channel and click on
Push Video Frame - Nothing happens
- Click on
Push Video Frameagain to make it work
Expected behavior
When calling pushVideoFrame, if the video publishing state is not published, SDK should automatically publish it and push the video frame accordingly.
I'm so sorry that we are on holiday until 02/17, it's better to raise a new ticket to Agora Support if you face an urgent issue.
@littleGnAl No rush, but please get back to this after holiday.
As I know the pushVideoFrame is not guaranteed to work for each call, it's the expected behavior of this API.
As I know the
pushVideoFrameis not guaranteed to work for each call, it's the expected behavior of this API.
But why have an API that does not guarantee to work though?
In case it does not work, no exceptions are thrown also.
The function itself did call success, but the video frame may be dropped, so you can not see it on the remote side.
The function itself did call success, but the video frame may be dropped, so you can not see it on the remote side.
That's the problem, how can we tell when the API is success but the frame is dropped?
From the sample project, it happens 100% on first try, but we can detect based on onVideoPublishStateChanged to call the API again. Since you said the API is not guarantee to success, how can we detect if more frames are dropped if we continue using the API? (e.g: Push another picture as video frame)
Maybe you can call the pushVideoFrame repeatedly until it receives the pubStatePublished of onVideoPublishStateChanged . Or use the pushEncodedVideoImage function.
pushEncodedVideoImage triggers onEncodedVideoFrameReceived immediately but we still need to render that frame on AgoraVideoView using pushVideoFrame.
I have done more tests and found that everytime we want to render a custom frame, we have to call pushVideoFrame twice for it to take effect. To reproduce, use the sample to push an image as custom frame, then continue pushing a different image, each time, it takes 2 pushVideoFrame call to take effect. I think this is a bug and needs improvement.
I'm so sorry that it's the expected behavior at this time, maybe improve in the future.
@littleGnAl I recently come across this API enableVideoImageSource, base on the API description, it should satisfy my requirement, however, I cannot find any example for this API in the sample project, I've been using it like this but somehow it's not working:
await agoraEngine.enableVideoImageSource(
enable: true,
options: ImageTrackOptions(
imageUrl: imageUrl,
mirrorMode: VideoMirrorModeType.videoMirrorModeDisabled,
),
);
From my understanding, imageUrl here is the path to the image file saved in the device, I have tried both network url and local file path but no luck, the Videoview only displays black frame.
This is the log that prints:
[info] [iris_rtc_engine_impl.cc:96] api name RtcEngine_enableVideoImageSource params {"enable":true,"options":{"imageUrl":"/data/user/0/sample.project.dev/cache/2-21-2024_11-3-50.png","fps":15,"mirrorMode":2}}
RtcEngine::enableVideoImageSource(this:0xb400007d4f19e000, {"enable":1, "options":{"imageUrl":"/data/user/0/sample.project.dev/cache/2-21-2024_11-3-50.png", "fps":30, "mirrorMode":2}})
...
[info] [iris_rtc_engine_impl.cc:112] api name RtcEngine_enableVideoImageSource ret 0 result {"result":0}
[VQC DEBUG] qp avg is19.000000, cur bit is:1111, cur fps is 30.000000, width 360
Based on our Native API example, the url should be an absolute local path
https://github.com/search?q=repo%3AAgoraIO%2FAPI-Examples%20enableVideoImageSource&type=code
Based on our Native API example, the url should be an absolute local path https://github.com/search?q=repo%3AAgoraIO%2FAPI-Examples%20enableVideoImageSource&type=code
The thing is, I called the API and it returns code == 0, which is success already, my path is also correct, but the API is not working.
I have tested on both Flutter and Android version (downloaded here https://docs.agora.io/en/sdks?platform=android), running the sample RTC Live Streaming and yet the native version does not work as well.
Please help me take a look at this, this really affects our development.
I tested it based on our example, and it seems to work fine, note that this API not work for local preview.
local:
remote:
@littleGnAl After further testing, I was able to adjust it to our requirement, so this is good enough for now. Thank you for your support
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please raise a new issue.