WebRTC: Off-sync audio/video in DVR files

[Iskander Shafikov]

Describe the bug
In SRS Stack 5, enabling DVR with any settings (including default) for the default vhost results in off-sync audio/video files when trying to record WebRTC streams. The problem is especially conspicuous for small video fragments (10 sec or less).

Version
SRS Stack 5 (latest) via Docker: ossrs/srs-stack:5

To Reproduce
Steps to reproduce the behavior:

1. Use the following bare-bones server config:

server_id           srs-server;
listen              1935;
max_connections     1000;
daemon              on;
disable_daemon_for_docker off;
utc_time            on;
ff_log_dir          /data/ffmpeg_logs;

rtc_server {
    enabled         on;
    listen          8000;          
    candidate       $CANDIDATE;
    protocol        udp;            
    reuseport       5;
    use_auto_detect_network_ip off;
    api_as_candidates off;    
}

vhost __defaultVhost__ {

    # General settings
    tcp_nodelay         on;
    min_latency         on;

    # WebRTC
    rtc {
        enabled         on;
        nack            on;
        twcc            on;
        rtmp_to_rtc     off;
        rtc_to_rtmp     on;
    }

    # Digital Video Recording
    dvr {
        enabled         on;
        dvr_apply       all;
        dvr_plan        session;
        dvr_path        /data/record/[app]/[stream].[timestamp].mp4;    # tried also ...flv, but no effect
        dvr_wait_keyframe       on;
        time_jitter             full;  # full / zero
    }
}

2. Create a WebRTC stream by publishing to https://MY-HOST.com/rtc/v1/whip/?app=live&stream=100&secret=SRS_PUB_SECRET
3. Go to your DVR path (/data/record/live) and download the resulting video(s), e.g. 100.TIMESTAMP.mp4
4. Observe off-sync audio/video.
5. Replace dvr_path in config for .....flv and do steps 2-5 above. Same issue remains.

Expected behavior
Expected fully-sync DVR files (mp4 or flv), with no data loss.

Additional context
For small videos, 10 sec or less, even video data is often lossy: the resulting DVR files get frozen or scrambled, with audio completely off-sync.

[Winlin]

Does this issues exists, when play the rtmp stream? You can play by ffplay or VLC, the url should be rtmp://MY-HOST.com/live/100 as such.

And, could you help to enable HLS, and check whether the HLS files is sync or not?

[Iskander Shafikov]

The HLS was in sync, but very, very slow. Actually, I couldn't manage to get HLS to play with satisfactory speed, so had to disable it. As to RTMP, I will check, but the live FLV streams are OK, video and audio is in sync. I also tried to switch to RTC via TCP, but it didn't help. There seems to be something wrong with transcoding from RTMP.

[Iskander Shafikov]

Update: I've checked RTMP - no sync issues found. So my previous guess appears right - something is wrong with DVR.

[Winlin]

After converting RTC to RTMP, DVR muxes RTMP to FLV or MP4 file.

If RTMP is synced and FLV/MP4 is not synced, the most possible root cause may be:

1. Your video stream content is special and causes this problem. Please try publishing by Chrome in your PC, and check whether it works well.
2. The DVR logical is incorrect. Please try to record the RTMP to FLV by FFmpeg, and check whether the file is ok.

BTW: What's your WebRTC client? Did you try Chrome, Firefox, or Safari? Did you try another different PC, like Mac or Windows? This can help us to reproduce the problem.

[Iskander Shafikov]

I use Chrome as client, but the problem persists even when using mobile clients (including Android Chrome and Apple Safari). As t recording to FLV, what code should I add to srs.conf to do that? I suppose, it should be somewhere in transcode?

[Winlin]

Not transcode of SRS, no need to change the config for SRS, you only need to record the RTMP by FFmpeg, for example:

ffmpeg -i rtmp://your-stream -c copy -f flv -y output.flv

Then you can check the output file.

[Iskander Shafikov]

Here's an example of off-sync DVR that I recorded just now.

livestream.1710817176050.1.mp4

[Iskander Shafikov]

As you can see, the audio is ahead of video.

[Iskander Shafikov]

Not transcode of SRS, no need to change the config for SRS, you only need to record the RTMP by FFmpeg, for example:

ffmpeg -i rtmp://your-stream -c copy -f flv -y output.flv

Then you can check the output file.

This method produces well-sync videos, I've checked both FLV and MP4 encoding. No issues found except for poor quality (maybe just my browser outputs a small picture size).

[Winlin]

Thank you, I believe this issue is in the DVR module, not about streaming or even HLS.

[Iskander Shafikov]

OK, so the bug is accepted? I've managed to make DVR using manual ffmpeg in the stream on_publish callback. I suggest making the same change in the SRS code. The ffmpeg command I use:

ffmpeg -i rtmp://<MY-STREAM> -async 1 -c:v copy -c:a copy -y OUTFILE.mp4

The -async 1 is required to cope with out-of-sync audio.

[Winlin]

@S0mbre Indeed, the bug is accepted, and I appreciate your assistance. :)

Regrettably, I cannot commit to a specific deadline for resolving this bug, as it depends on the developers' time availability and interest. In summary, due to my lack of time and interest in this issue, I will not be rectifying it.

Everyone is welcome to address this bug, and if you wish to do so, please submit a pull request. The reproduction steps and background information on this issue are crucial, so thank you for clarifying this bug.