After pushing the audio and video data streams, when pushing pure video data to this channel, VLC player will have abnormal playback and the media information will display an additional audio codec information.

Question

After pushing the audio and video data streams, when pushing pure video data to this channel, VLC player will have abnormal playback and the media information will display an additional audio codec information.

limjoe opened this issue 5 years ago · comments

Description'

Please ensure that you maintain the markdown structure.

It seems that SRS caches the audio meta information of a specific channel. After pushing a data stream containing both audio and video to a channel, if a data stream containing only video is pushed later, VLC player will play the video abnormally without black screen, and VLC media information will display an additional audio codec information.'

Please ensure that you maintain the markdown structure.

SRS version: 3.0.121
The log of SRS is as follows:

Please ensure that you maintain the markdown structure.

[2020-03-06 16:33:40.357][Trace][23607][469] RTMP client ip=118.114.22.175, fd=10
[2020-03-06 16:33:40.548][Trace][23607][469] simple handshake success.
[2020-03-06 16:33:40.550][Trace][23607][469] connect app, tcUrl=rtmp://120.92.213.203:11935/live, pageUrl=, swfUrl=, schema=rtmp, vhost=120.92.213.203, port=11935, app=l
ive, args=null
[2020-03-06 16:33:40.550][Trace][23607][469] protocol in.buffer=0, in.ack=0, out.ack=0, in.chunk=128, out.chunk=128
[2020-03-06 16:33:40.653][Trace][23607][469] client identified, type=fmle-publish, vhost=120.92.213.203, app=live, stream=LIVEPL16DD80CE047_2, param=, duration=0ms
[2020-03-06 16:33:40.653][Trace][23607][469] connected stream, tcUrl=rtmp://120.92.213.203:11935/live, pageUrl=, swfUrl=, schema=rtmp, vhost=__defaultVhost__, port=11935
, app=live, stream=LIVEPL16DD80CE047_2, param=, args=null
[2020-03-06 16:33:40.653][Trace][23607][469] source url=/live/LIVEPL16DD80CE047_2, ip=118.114.22.175, cache=0, is_edge=0, source_id=-1[-1]
[2020-03-06 16:33:40.959][Trace][23607][469] ignore disabled exec for vhost=__defaultVhost__
[2020-03-06 16:33:40.959][Trace][23607][469] set fd=10 TCP_NODELAY 0=>1
[2020-03-06 16:33:40.959][Trace][23607][469] start publish mr=0/350, p1stpt=20000, pnt=5000, tcp_nodelay=1
[2020-03-06 16:33:41.011][Trace][23607][469] got metadata, width=1920, height=1080, vcodec=7
[2020-03-06 16:33:41.012][Trace][23607][469] 36B video sh,  codec(7, profile=Baseline, level=Other, 1920x1088, 0kbps, 0.0fps, 0.0s)

[2020-03-06 16:33:52.891][Trace][23607][470] RTMP client ip=118.114.22.175, fd=11
[2020-03-06 16:33:52.941][Trace][23607][470] complex handshake success
[2020-03-06 16:33:52.986][Trace][23607][470] connect app, tcUrl=rtmp://120.92.213.203:11935/live, pageUrl=, swfUrl=, schema=rtmp, vhost=120.92.213.203, port=11935, app=l
ive, args=null
[2020-03-06 16:33:52.986][Trace][23607][470] protocol in.buffer=0, in.ack=0, out.ack=0, in.chunk=128, out.chunk=128
[2020-03-06 16:33:53.258][Trace][23607][470] ignore AMF0/AMF3 command message.
[2020-03-06 16:33:53.668][Trace][23607][470] ignore AMF0/AMF3 command message.
[2020-03-06 16:33:53.668][Trace][23607][470] client identified, type=Play, vhost=120.92.213.203, app=live, stream=LIVEPL16DD80CE047_2, param=, duration=-1ms
[2020-03-06 16:33:53.668][Trace][23607][470] connected stream, tcUrl=rtmp://120.92.213.203:11935/live, pageUrl=, swfUrl=, schema=rtmp, vhost=__defaultVhost__, port=11935, app=live, stream=LIVEPL16DD80CE047_2, param=, args=null
[2020-03-06 16:33:53.668][Trace][23607][470] source url=/live/LIVEPL16DD80CE047_2, ip=118.114.22.175, cache=0, is_edge=0, source_id=469[469]
[2020-03-06 16:33:53.669][Trace][23607][470] set fd=11 TCP_NODELAY 0=>1
[2020-03-06 16:33:53.669][Trace][23607][470] dispatch cached gop success. count=0, duration=0
[2020-03-06 16:33:53.669][Trace][23607][470] create consumer, active=1, queue_size=0.00, jitter=10000000
[2020-03-06 16:33:53.669][Trace][23607][470] set fd=11, SO_SNDBUF=46080=>50000, buffer=100ms
[2020-03-06 16:33:53.669][Trace][23607][470] start play smi=0ms, mw_sleep=100, mw_enabled=1, realtime=1, tcp_nodelay=1

[2020-03-06 16:34:04.028][Trace][23607][470] -> PLA time=10000023, msgs=2, okbps=374,0,0, ikbps=2,0,0, mw=100
[2020-03-06 16:34:05.959][Trace][23607][469] <- CPB time=20000027, okbps=1,0,0, ikbps=409,0,0, mr=0/350, p1stpt=20000, pnt=5000
[2020-03-06 16:34:13.699][Trace][23607][470] -> PLA time=20000086, msgs=2, okbps=399,0,0, ikbps=1,0,0, mw=100
[2020-03-06 16:34:15.959][Trace][23607][469] <- CPB time=30000114, okbps=0,1,0, ikbps=417,408,0, mr=0/350, p1stpt=20000, pnt=5000
[2020-03-06 16:34:24.620][Trace][23607][470] -> PLA time=31000089, msgs=1, okbps=409,407,0, ikbps=0,0,0, mw=100
[2020-03-06 16:34:30.959][Trace][23607][469] <- CPB time=45000199, okbps=0,1,0, ikbps=421,408,0, mr=0/350, p1stpt=20000, pnt=5000
[2020-03-06 16:34:34.885][Trace][23607][470] -> PLA time=41000178, msgs=2, okbps=416,407,0, ikbps=0,0,0, mw=100
[2020-03-06 16:34:44.951][Trace][23607][470] -> PLA time=51000357, msgs=2, okbps=420,407,0, ikbps=0,0,0, mw=100
[2020-03-06 16:34:45.959][Trace][23607][469] <- CPB time=60000378, okbps=0,0,0, ikbps=422,441,0, mr=0/350, p1stpt=20000, pnt=5000
[2020-03-06 16:34:54.756][Trace][23607][470] -> PLA time=61000388, msgs=2, okbps=419,429,0, ikbps=0,0,0, mw=100
[2020-03-06 16:34:55.959][Trace][23607][469] <- CPB time=70000384, okbps=0,0,0, ikbps=424,441,0, mr=0/350, p1stpt=20000, pnt=5000
[2020-03-06 16:35:04.846][Trace][23607][470] -> PLA time=71000574, msgs=2, okbps=425,429,0, ikbps=0,0,0, mw=100
[2020-03-06 16:35:05.959][Trace][23607][469] <- CPB time=80000559, okbps=0,0,0, ikbps=426,441,0, mr=0/350, p1stpt=20000, pnt=5000
[2020-03-06 16:35:14.831][Trace][23607][470] -> PLA time=81000598, msgs=2, okbps=422,429,0, ikbps=0,0,0, mw=100
[2020-03-06 16:35:15.959][Trace][23607][469] <- CPB time=90000600, okbps=0,0,0, ikbps=426,426,0, mr=0/350, p1stpt=20000, pnt=5000
[2020-03-06 16:35:24.765][Trace][23607][470] -> PLA time=91000729, msgs=2, okbps=422,429,0, ikbps=0,0,0, mw=100
[2020-03-06 16:35:25.959][Trace][23607][469] <- CPB time=100000706, okbps=0,0,0, ikbps=428,426,0, mr=0/350, p1stpt=20000, pnt=5000
[2020-03-06 16:35:35.959][Trace][23607][469] <- CPB time=110000734, okbps=0,0,0, ikbps=427,426,0, mr=0/350, p1stpt=20000, pnt=5000
[2020-03-06 16:35:38.592][Trace][23607][469] shrinking, size=0, removed=120, max=10000ms
[2020-03-06 16:35:45.959][Trace][23607][469] <- CPB time=120000762, okbps=0,0,0, ikbps=427,427,0, mr=0/350, p1stpt=20000, pnt=5000
[2020-03-06 16:35:48.864][Trace][23607][469] shrinking, size=0, removed=121, max=10000ms
[2020-03-06 16:35:55.959][Trace][23607][469] <- CPB time=130000872, okbps=0,0,0, ikbps=428,427,0, mr=0/350, p1stpt=20000, pnt=5000
[2020-03-06 16:35:58.597][Error][23607][470][4] serve error code=1011 : service cycle : rtmp: stream service : rtmp: send 2 messages : send messages : writev : writev timeout 30000 ms
thread [23607][470]: do_cycle() [src/app/srs_app_rtmp_conn.cpp:210][errno=4]
thread [23607][470]: service_cycle() [src/app/srs_app_rtmp_conn.cpp:400][errno=4]
thread [23607][470]: do_playing() [src/app/srs_app_rtmp_conn.cpp:786][errno=62]
thread [23607][470]: send_and_free_messages() [src/protocol/srs_rtmp_stack.cpp:834][errno=62]
thread [23607][470]: srs_write_large_iovs() [src/protocol/srs_protocol_utility.cpp:341][errno=62]
thread [23607][470]: writev() [src/service/srs_service_st.cpp:585][errno=62](Interrupted system call)
[2020-03-06 16:36:05.959][Trace][23607][469] <- CPB time=140000940, okbps=0,0,0, ikbps=428,427,0, mr=0/350, p1stpt=20000, pnt=5000
[2020-03-06 16:36:15.959][Trace][23607][469] <- CPB time=150001137, okbps=0,0,0, ikbps=429,431,0, mr=0/350, p1stpt=20000, pnt=5000
[2020-03-06 16:36:25.959][Trace][23607][469] <- CPB time=160001205, okbps=0,0,0, ikbps=428,431,0, mr=0/350, p1stpt=20000, pnt=5000
[2020-03-06 16:36:35.959][Trace][23607][469] <- CPB time=170001261, okbps=0,0,0, ikbps=428,431,0, mr=0/350, p1stpt=20000, pnt=5000

[2020-03-06 16:36:45.959][Trace][23607][469] <- CPB time=180001383, okbps=0,0,0, ikbps=429,430,0, mr=0/350, p1stpt=20000, pnt=5000
[2020-03-06 16:36:55.959][Trace][23607][469] <- CPB time=190001417, okbps=0,0,0, ikbps=428,430,0, mr=0/350, p1stpt=20000, pnt=5000
[2020-03-06 16:37:05.959][Trace][23607][469] <- CPB time=200001499, okbps=0,0,0, ikbps=430,430,0, mr=0/350, p1stpt=20000, pnt=5000

The configuration of SRS is as follows:

Please ensure that you maintain the markdown structure.

listen              1935;
max_connections     1000;
daemon              off;
srs_log_tank        console;
grace_start_wait    700;
grace_final_wait    800;

http_server {
    enabled         on;
    listen          18080;
    dir             ./objs/nginx/html;
}

stats {  
    network         0;   
    disk            vda vdb;
}

http_api {  
    enabled         on;   
    listen          1985;  
    crossdomain     on;
}

vhost __defaultVhost__ {
    min_latency     on;
    tcp_nodelay     on;

    publish {
        mr off;
    }

    play {
        gop_cache       off;
        queue_length    10;
        mw_latency      100;
    }    
}

Replay

Please ensure that you maintain the markdown structure.

How to replay bug?
The pure video data file used for replay can be found at: link (password: cnvp).
Please ensure that you maintain the markdown structure.

1. Pushing pure video stream data to channel A using ffmpeg, the rtmp playback is normal, and VLC's meta information also shows that only the video type stream is present.
Please ensure that you maintain the markdown structure.
2. ffmpeg -re -i 1583334009334.flv -vcodec copy -f flv -y rtmp://localhost:1935/live/LIVEPL16DD80CE047_2
1. Whether using ffmpeg or pushing data containing both audio and video to ChannelA, the rtmp playback is normal, and VLC's meta information will display two streams, one for audio and one for video.
Please ensure that you maintain the markdown structure.
2. ffmpeg -re -i 1583334009334.flv -vcodec copy -acodec copy -f flv -y rtmp://localhost:1935/live/LIVEPL16DD80CE047_2
1. Repeating the operation in step 1, pushing a stream of pure video data to Channel A results in abnormal rtmp playback (VLC cannot play, it remains black after a long buffering time). In VLC's meta information, besides displaying the video's meta information, an additional audio stream meta information will also be displayed.
2. Using ffmpeg to push the stream of pure video to a new channel B, rtmp playback is normal again.
3. Suspected that VLC or ffprobe may cache meta information, but even after closing and reopening VLC, the issue remains as described above.
4. Only after restarting the SRS service, the additional display of audio meta information will no longer occur.
Please ensure that you maintain the markdown structure.

Expected behavior

Expected behavior:

When pushing pure video data after previously pushing data containing audio and video, VLC and other web clients should be able to support playback without returning additional audio meta information.

TRANS_BY_GPT3

Qiao Lin · Answer 1 · Tue Mar 10 2020 23:18:42 GMT+0800 (China Standard Time)

Looking forward to the author taking a look at this issue, thank you.
This issue occurs when muting the device and then re-enabling the microphone, causing playback issues in Web player SDKs like AliPlayer.

TRANS_BY_GPT3

Winlin · Answer 2 · Thu Mar 12 2020 10:36:27 GMT+0800 (China Standard Time)

Attempt with only video scenario, conclusion when video changes:

Player does not disconnect, VLC fails to play, Flash works normally.
After player disconnects and replays, both VLC and Flash work normally.

Below are the steps:

Start SRS.
Only push video stream, encoded as 1500Kbps, 512x400, 25fps, baseline, without audio: ffmpeg -re -i doc/source.200kbps.768x320.flv -vcodec libx264 -s 512x400 -b:v 1500k -r 25 -profile:v baseline -an -f flv -y rtmp://localhost/live/livestream
Only push video stream, encoded as 600kbps, 304x208, 20fps, high: ffmpeg -re -i doc/source.200kbps.768x320.flv -vcodec libx264 -s 304x208 -b:v 600k -r 20 -profile:v high -an -f flv -y rtmp://localhost/live/livestream

FFMPEG, VLC, and Flash can all successfully pull the video. This shows that there is no problem with the player restarting playback after changing the video.

Note that if VLC is not disconnected, changing the codec and video size during streaming will result in a distorted screen.

The first time VLC is like this:

After the second streaming, VLC (if not disconnected) is like this:

After disconnecting VLC and playing, it is normal.

It can be seen that if the playback is not disconnected, VLC will have screen flickering after changing the video, while Flash is normal.

TRANS_BY_GPT3

Winlin · Answer 3 · Thu Mar 12 2020 10:49:01 GMT+0800 (China Standard Time)

Trying with both audio and video, keeping the codec unchanged, changing from audio and video to pure video, the conclusions are as follows:

The player does not disconnect, Flash works fine, but VLC takes a long time to play. This is because SRS caches the audio and needs to be fixed.
When the player disconnects and replays, Flash works fine, but VLC still takes a long time. This needs to be fixed.

Below are the steps:

Start SRS.
Push audio and video streams, copy video, copy audio: ffmpeg -re -i doc/source.200kbps.768x320.flv -vcodec copy -acodec copy -f flv -y rtmp://localhost/live/livestream
Only push video stream, copy video, disable audio: ffmpeg -re -i doc/source.200kbps.768x320.flv -vcodec copy -an -f flv -y rtmp://localhost/live/livestream

At this time, VLC playback fails while Flash playback succeeds.

Using srs_rtmp_dump to view server data:

[root@093b9be1d67d trunk]# ./objs/research/librtmp/srs_rtmp_dump -r rtmp://127.0.0.1:1935/live/livestream
[T][331][2020-03-12 03:13:29.963] Audio packet id=6/50.8/19.7, type=Audio, dts=0, pts=0, ndiff=2, diff=0, size=4, AAC(44KHz,16bit,Stereo,SH), (
0xaf 0x00 0x12 0x10 )
[T][331][2020-03-12 03:13:29.963] Video packet id=7/44.9/22.3, type=Video, dts=0, pts=0, ndiff=1, diff=0, size=46, H.264(SH,I), (0x17 0x00 0x00
 0x00 0x00 0x01 0x64 0x00)
[T][331][2020-03-12 03:13:29.963] Video packet id=8/40.4/24.8, type=Video, dts=0, pts=80, ndiff=5, diff=0, size=5137, H.264(Nalu,I), (0x17 0x01
 0x00 0x00 0x50 0x00 0x00 0x02)

It can be observed that the SequenceHeader of the audio is cached. At this point, since there is no audio available, it causes VLC to fail in synchronizing the audio and video.

If you wait long enough (around 10 seconds), the video can be played.

TRANS_BY_GPT3

Qiao Lin · Answer 4 · Thu Mar 12 2020 10:52:37 GMT+0800 (China Standard Time)

Well, but in actual usage, the waiting time for opening the live broadcast is too long, which will result in a poor user experience or be considered a bug.

TRANS_BY_GPT3

Winlin · Answer 5 · Thu Mar 12 2020 11:06:38 GMT+0800 (China Standard Time)

In the case of pure audio, changing the encoding alters the behavior of the player. Conclusion:

The player does not disconnect, Flash works normally, but VLC audio may have anomalies.
When the player disconnects and replays, both Flash and VLC work normally.

The steps are as follows:

Start SRS.
Only push audio stream, no video. Audio codec is AAC, 44100 sample rate, 2 channels, 100kbps: ffmpeg -re -i doc/source.200kbps.768x320.flv -vn -acodec libfdk_aac -ar 44100 -ac 2 -b:a 100k -f flv -y rtmp://localhost/live/livestream
Only push audio stream, no video. Audio codec is AAC, 32000 sample rate, 1 channel, 64kbps: ffmpeg -re -i doc/source.200kbps.768x320.flv -vn -acodec libfdk_aac -ar 32000 -ac 1 -b:a 64k -f flv -y rtmp://localhost/live/livestream

FFMPEG, VLC, and Flash can all successfully pull the stream. This indicates that there is no issue with the player restarting after changing the audio.

Note that if VLC is not disconnected, changing the codec during streaming can result in audio abnormalities.

The first time VLC is like this:

After the second streaming, VLC (if not disconnected) looks like this, but the audio is distorted:

After disconnecting VLC and playing, it is normal.

It can be seen that if the playback is not disconnected, VLC will have audio abnormalities after changing the audio encoding, while Flash is normal.

TRANS_BY_GPT3

Winlin · Answer 6 · Thu Mar 12 2020 11:10:34 GMT+0800 (China Standard Time)

In conclusion, the conclusion is as follows:

When the streaming encoder's audio or video encoding/decoding parameters remain unchanged, only disabling or enabling a stream:
- If the player remains connected, Flash works normally. VLC caches the previous headers and takes a longer time to recover.
- If the player disconnects and reconnects, Flash works normally. VLC works normally. SRS does not cache the previous headers. Refer to 4b395f6
When the streaming encoder's audio or video encoding/decoding parameters are changed and the stream is not disabled or enabled:
- If the player remains connected, Flash works normally. VLC does not support changes in encoding/decoding information and may have exceptions.
- If the player disconnects and reconnects, Flash works normally. VLC works normally.

TRANS_BY_GPT3

Winlin · Answer 7 · Thu Mar 12 2020 11:53:05 GMT+0800 (China Standard Time)

SRS cache video/audio sequence header has a purpose to avoid sending duplicate SequenceHeader to the same client. For example, if the player remains unchanged but the encoder re-pushes with the same encoding parameters, some players may encounter issues if the encoding and decoding information is directly sent to the client.

    // whether consumer should drop for the duplicated sequence header.
    bool drop_for_reduce = false;
    if (is_sequence_header && meta->vsh() && _srs_config->get_reduce_sequence_header(req->vhost)) {
        if (meta->vsh()->size == msg->size) {
            drop_for_reduce = srs_bytes_equals(meta->vsh()->payload, msg->payload, msg->size);
            srs_warn("drop for reduce sh video, size=%d", msg->size);
        }
    }

We need to separate these two functions.

Use previous_vsh and previous_ash to determine if it is a duplicate SequenceHeader and update the previous information when unpublish and update_vsh and update_ash are executed.
Clear vsh and ash each time publish is executed, but do not clear the previous information. This way, the new player will not cache the previous stream information.

    play {
        reduce_sequence_header  on;
    }

After verification, this feature is also working fine.

TRANS_BY_GPT3

Winlin · Answer 8 · Thu Mar 12 2020 12:08:39 GMT+0800 (China Standard Time)

Do not change the encoding header, especially for VLC testing, as follows:

Note: Changing the encoding header will definitely cause problems with VLC. We will no longer test it as this is an issue with VLC.

Disable stream

Push audio and video streams, then only push the video stream, and then only push the audio stream to observe VLC behavior.

Push audio and video streams, VLC waits for about 3 seconds to play:
ffmpeg -ss 60s -re -i doc/source.200kbps.768x320.flv -vcodec copy -acodec copy -f flv -y rtmp://localhost/live/livestream
Only push the audio stream, VLC recovers after about 20 seconds:
ffmpeg -ss 60s -re -i doc/source.200kbps.768x320.flv -vn -acodec copy -f flv -y rtmp://localhost/live/livestream
Only push the video stream, VLC recovers after about 20 seconds:
ffmpeg -ss 60s -re -i doc/source.200kbps.768x320.flv -vcodec copy -an -f flv -y rtmp://localhost/live/livestream
Push audio and video streams, VLC recovers quickly:
ffmpeg -ss 60s -re -i doc/source.200kbps.768x320.flv -vcodec copy -acodec copy -f flv -y rtmp://localhost/live/livestream

Note: VLC takes a longer time to recover when disabling streams because it has a larger cache. For example, when switching from an audio and video stream to a pure video stream, there will still be a certain amount of cache time.

New Video Stream

Push the audio stream, then push the audio-video stream, and observe VLC behavior:

Only push the audio stream, VLC starts playing after about 10 seconds: ffmpeg -ss 60s -re -i doc/source.200kbps.768x320.flv -vn -acodec copy -f flv -y rtmp://localhost/live/livestream
Push the audio-video stream, VLC recovers quickly but without video: ffmpeg -ss 60s -re -i doc/source.200kbps.768x320.flv -vcodec copy -acodec copy -f flv -y rtmp://localhost/live/livestream

Note: It can be seen that VLC relies on the initial encoding information from the first playback. If there is no video, even if there is video information later on, the video still cannot be seen.

Replaying allows you to see the audio and video streams.

Add audio stream

Push video stream, then push audio and video stream, observe VLC behavior:

Only push video stream, VLC starts playing after about 10 seconds:
ffmpeg -ss 60s -re -i doc/source.200kbps.768x320.flv -vcodec copy -an -f flv -y rtmp://localhost/live/livestream
Push audio and video stream, VLC recovers quickly but without audio:
ffmpeg -ss 60s -re -i doc/source.200kbps.768x320.flv -vcodec copy -acodec copy -f flv -y rtmp://localhost/live/livestream

Note: Same as above, audio cannot be heard. VLC relies on the codec information obtained during the first time.

Repeated Sequence Header

VLC has no problem with repeated Sequence Headers. It can handle repeated ones, but not different ones.

Note: Flash also has no problem with repeated Sequence Headers.

TRANS_BY_GPT3

Winlin · Answer 9 · Thu Mar 12 2020 12:09:23 GMT+0800 (China Standard Time)

In summary: VLC relies on the initial encoding header information. Although SRS provides the correct encoding header, it still cannot correctly recognize the newly added stream.

TRANS_BY_GPT3