Access to Raw Video Data

[rgb1380]

Question:
How do we get access to raw video stream in Flutter. I am looking for the equivalent of this in the native SDK:
https://docs.agora.io/en/Interactive%20Broadcast/raw_data_video_android?platform=Android

[Arthas]

For performance reasons, we do not provide this function in dart layer for the time being. If you need to use it, you can fork our repository and add it to the native layer.
Here is the doc.
https://docs.agora.io/en/Interactive%20Broadcast/raw_data_video_android?platform=Android
https://docs.agora.io/en/Interactive%20Broadcast/raw_data_video_apple?platform=iOS

[rgb1380]

What's the concern with performance? Have you tried this and found the performance to be unacceptable?

[Arthas]

the byte array from c++ to java, from java to dart is not a good way to process raw data.
You can describe your requirements and what to do with raw data in dart layer.

[rgb1380]

What do mean by

You can describe your requirements and what to do with raw data in dart layer.

[Arthas]

We will collect customer requirements in this area and then see if we do some plug-ins in the native layer to implement them according to the actual situation.

[rgb1380]

We intend to use the video feed for a more advanced face-detection than is available with the agora SDK. The SDK face-detection provides the location of the face but we also need to extract location of facial features such as eyes, nose, etc.

[Arthas]

Raw data plugin is beta now, you can refer to the example and implement the code in the Android&iOS or C++ layer.

[yaizudamashii]

I'd like to use Agora audio chat with just_audio so I can support background play/pause. In order to do it, I would need audio data as raw bytes, or as an url, if we can somehow access channel's url. Is there a way to get audio data in one of those formats in Agora's Flutter SDK?

[Arthas]

Raw data plugin is beta now, you can refer to the example and implement the code in the Android&iOS or C++ layer.

@yaizudamashii here

[Kalana Perera]

Hi @LichKing-2234,

Sorry this is not related to Agora SDK, currently, I am using on this https://github.com/flutter-webrtc/flutter-webrtc (Not the creator)

We also need to access raw video data,

With your experience, would you be able to guide us on the correct path?

[Nikita Lavrov]

the byte array from c++ to java, from java to dart is not a good way to process raw data.

FYI @LichKing-2234, there are plenty of use-cases where you don't actually need to process raw data in any way whatsoever, you just need to set up a callback function to be triggered whenever a new frame is encoded. That's exactly how I use the plugin currently:

android/src/main/java/io/agora/rtc/rawdata/base/IVideoFrameObserver.java

  public int getObservedFramePosition() {
    return POSITION_PRE_ENCODER | POSITION_PRE_RENDERER;
  }

android/src/main/kotlin/io/agora/agora_rtc_rawdata/AgoraRtcRawdataPlugin.kt

"registerVideoFrameObserver" -> {
        if (videoObserver == null) {
          videoObserver = object : IVideoFrameObserver((call.arguments as Number).toLong()) {
            override fun onPreEncodeVideoFrame(videoFrame: VideoFrame): Boolean {
              uiThreadHandler.post {
                channel.invokeMethod("onFrame", System.currentTimeMillis())
              }
              return true
            }
          }
        }
        videoObserver?.registerVideoFrameObserver()
        result.success(null)
      }

Furthermore, however inefficient it is to process data by passing it to the Dart layer, in some situations it might be an acceptable and optimal trade-off. If you know you will only ever stream 1-2 FPS, it would probably be totally acceptable to process frames in Dart. I haven't tried that, but if it works at all, I can see the benefit in doing so to avoid dealing with Swift/Java and C++.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please raise a new issue.