webrtc VideoFrame I420 not compatible with

[kasperdun]

Hi, webrtc in your repo io.github.crow-misia.libmediasoup-android:libmediasoup-android
has VideoFrame.Buffer and this one has method toI420() that returns VideoFrame.I420Buffer
But VideoFrame.I420Buffer is not compatible with I420Buffer from libyuv-android repo.

Can you fix it?
May be add methods to convert from webrtc I420 to your I420 and back from your I420 to webrtc I420

I have VideoProcessor that has callback fun onFrameCaptured(inputVideoFrame: VideoFrame?) and I need change this frame every frame.

Now I have to convert VideoFrame.Buffer.toI240() to nv21ByteArray : ByteArray - I have big function for that with code which I don't understand.

Then convert nv21 ByteArray to ByteBuffer, else I can't convert it to your buffers

byteBuffer = ByteBuffer.allocateDirect(nv21ByteArray.size).also {
                it.put(nv21ByteArray)
            }

Then I wrap byteBuffer by your Nv21Buffer

var nv21Buffer: Nv21Buffer = Nv21Buffer.wrap(byteBuffer, frame.buffer.width, frame.buffer.height)

Then I convert nv21Buffer to AbgrBuffer, then convert it to bitmap and do blur by Toolkit from com.google.android.renderscript
And now I need convert blured bitmap back to VideoFrame

            val abgrBuffer: AbgrBuffer = AbgrBuffer.allocate( frame.buffer.width, frame.buffer.height)

            nv21Buffer.convertTo(abgrBuffer)

            val inputBitmap = abgrBuffer.asBitmap()
            Toolkit.blur(inputBitmap, 20).also {
                it.copyPixelsToBuffer(abgrBuffer.asBuffer())
                it.recycle()
            }

            abgrBuffer.convertTo(nv21Buffer)

            val outFrame = VideoFrame(getFrameBufferByNv21Buffer(nv21Buffer), frame.rotation, frame.timestampNs)

            byteBuffer?.clear()
            byteBuffer = null

            nv21Buffer.close()
            abgrBuffer.close()

            return outFrame

This way so hard and I have too much memory leaks with buffers and bitmaps, maybe you can help me with this.
Thanks.

[Zenichi Amano]

This is not a bug.

Android I420 is formatted as I420, YV12, NV12, or NV21, depending on the device.

ByteBuffer.allocateDirect will not release memory until after GC, so it is no surprise that allocating memory on a frame-by-frame basis will result in insufficient memory.

[kasperdun]

@crow-misia Ok, maybe you know how I should do it?
Convert VideoFrame.I420Buffer to your buffer without memory leak?

[cguentherTUChemnitz]

This is not a bug.

Android I420 is formatted as I420, YV12, NV12, or NV21, depending on the device.

ByteBuffer.allocateDirect will not release memory until after GC, so it is no surprise that allocating memory on a frame-by-frame basis will result in insufficient memory.

How do we get than the information which format is used internally? For example when i use camerax capture use case on galaxy S8 it seems to report the image.getFormat() does provide me the information, that this is YUV_420_888, but to convert it correctly i have to use nv21 instead. Does anyone know how to robustly determine the correct image format here?

[kasperdun]

I reworked my solution, @crow-misia your library super useful thanks.