WebRTC server stuck with "pipeline full"

[Amir Zviran]

Hi Dustin,

I'm using the video-viewer to create a WebRTC server on my Jetson AGX. The Jetson has L4T version 32.7.1, Jetpack 4.6.1.
I'm running the following pipeline inside the dustynv/jetson-inference:r32.7.1 docker container:

video-viewer csi://0 webrtc://@:8554/output --ssl-key=key.pem --ssl-cert=cert.pem --bitrate=10000000 --output-codec=v4l2

Everything works fine at the beginning, but if my client disconnects and re-connects multiple times, eventually the server stops outputting video, and I see the following prints in the log:

[gstreamer] gstEncoder -- pipeline full, skipping frame 1800 (1280x720, 1382400 bytes)
video-viewer: captured 1800 frames (1280x720)
[gstreamer] gstEncoder -- pipeline full, skipping frame 1825 (1280x720, 1382400 bytes)
video-viewer: captured 1825 frames (1280x720)
[gstreamer] gstEncoder -- pipeline full, skipping frame 1850 (1280x720, 1382400 bytes)
video-viewer: captured 1850 frames (1280x720)
[gstreamer] gstEncoder -- pipeline full, skipping frame 1875 (1280x720, 1382400 bytes)
video-viewer: captured 1875 frames (1280x720)
[gstreamer] gstEncoder -- pipeline full, skipping frame 1900 (1280x720, 1382400 bytes)
video-viewer: captured 1900 frames (1280x720)
[gstreamer] gstEncoder -- pipeline full, skipping frame 1925 (1280x720, 1382400 bytes)
video-viewer: captured 1925 frames (1280x720)
...

I suspect it has something to do with the timings of stuff right after the client's connection.

Any idea why it happens and how it may be solved?

Thanks a lot!
Amir

[Amir Zviran]

@dusty-nv I'm still experiencing this issue, and I was wondering if there have been any updates or if you had a chance to look into it. Your guidance or insights on this matter would be immensely helpful.
Thanks!

[Dustin Franklin]

@amir-zviran unfortunately not, I think it's fair to say that the WebRTC plugins and connection process in GStreamer are "complicated" and unsure how to fully resolve this issue without really digging into the weeds for it, which I don't have the resources to do at the moment. Sorry about that.

[Amir Zviran]

@dusty-nv Thanks for the update and your honesty. I understand that it can be a complex issue. If you ever find the time or resources to delve into it, I'd greatly appreciate any further insights. In the meantime, I'll continue exploring potential solutions. Thanks again!

[ThorntonRobotic]

@dusty-nv I've been playing around a bit with this issue and have some observations. I'm running a webrtc server app in a Docker container on Jetson Xavier. Startup "./webRTCapp --ssl-key=server.pem --ssl-cert=server.pem /dev/video4 webrtc://@:8554/lopar". I have a Realsense D435i camera addressed by /dev/video4.
I have another container hosting httpd that's serving webpages to connect to webrtc server.
The webrtc server always works when I'm accessing through Chrome on my Windows PC (Windows 10, pretty powerful workstation). On my Macbook Air (M2) Chrome it never works(or at least almost never), I get the "pipeline full" error.
I did a little debugging and the symptom is that at least in my case in file gstEncoder.cpp:
// Sets up need-data callback
g_signal_connect(appsrcElement, "need-data", G_CALLBACK(onNeedData), this);

// When system fails with "pipeline full " this onNeedData callback function is never called and mNeedData = false
void gstEncoder::onNeedData( GstElement* pipeline, guint size, gpointer user_data ), and therefore
...
enc->mNeedData = true;
...

// My workaround was removing the return based on mNeedData flag
bool gstEncoder::encodeYUV( void* buffer, size_t size )
...
if( !mNeedData )
{
if( mOptions.frameCount % 25 == 0 )
LogVerbose(LOG_GSTREAMER "gstEncoder -- pipeline full, skipping frame %zu (%ux%u, %zu bytes)\n", mOptions.frameCount, mOptions.width, mOptions.height, size);

	// *** return true; // Commented out this line

...
// Continue processing YUV frame

I understand the reason for the check but wonder whether this is required since it seems there is a some race condition perhaps in the media negotiation that's causing the callback to be previously satisfied or not required.

A few thoughts on workaround, one would be to loosen this mNeedData check based on some mOptions parameter. The more optimal would be to understand root cause on why this callback is not executed. The former is pretty simple, the later is a more involved effort.

I'll spend some time to try to understand root cause but wanted to hear your thoughts before I spent too much time on this. If you want I can create a fork and make the workaround changes.

So far my testing with this patch always works on both Windows and Mac via Chrome.

[Dustin Franklin]

Interesting @ThorntonRobotic, thanks yes if it doesn't create gstreamer errors to continue pushing appsrc buffers even when the pipeline is "full", then I would use that as a workaround. All the gstwebrtc stuff and networking is tricky to debug and a complex gstreamer scenario, and it behaviors also change across versions, so I would not expect to track everything down and make it perfectly reliable. It does however utilize the hardware codecs.