Session crashes when processing malformed SDP
pabloFuente opened this issue · comments
Describe the bug
An OpenVidu Session seems to crash when processing a malformed SDP.
Expected behavior
Session should not crash. The request to publish a Stream with a malformed SDP should simply return an error to the client performing the request.
Wrong current behavior
The Session processes the request with the wrong SDP, and seems to bring the following unwanted behvior:
- Various errors in server and client logs
- Every participant’s video streams got stuck
- The session could not be properly closed (it became a ghost session, required OpenVidu server reboot to clear it)
OpenVidu tutorial where to replicate the error
openvidu-android seems to be the best place to start.
This is an SDP that supposedly triggers the error:
v=0
o=- 7719772216287633120 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1
a=msid-semantic: WMS
m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 102 0 8 106 105 13 110 112 113 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:c2WV
a=ice-pwd:GZ7nmy4N6x4Ca63YsFIPmJjj
a=ice-options:trickle renomination
a=fingerprint:sha-256 [REMOVED]
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:6 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=sendonly
a=msid:- 101
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:9 G722/8000
a=rtpmap:102 ILBC/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:112 telephone-event/32000
a=rtpmap:113 telephone-event/16000
a=rtpmap:126 telephone-event/8000
a=ssrc:2314895369 cname:9Vpk6URscoMETSuX
a=ssrc:2314895369 msid:- 101
a=ssrc:2314895369 mslabel:-
a=ssrc:2314895369 label:101
m=video 9 UDP/TLS/RTP/SAVPF 0
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:c2WV
a=ice-pwd:[REMOVED]
a=ice-options:trickle renomination
a=fingerprint:sha-256 [REMOVED]
a=setup:actpass
a=mid:1
a=extmap:14 urn:ietf:params:rtp-hdrext:toffset
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:13 urn:3gpp:video-orientation
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:12 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:11 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:6 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=sendonly
a=msid:- 100
a=rtcp-mux
a=rtcp-rsize
a=ssrc:4108384921 cname:9Vpk6URscoMETSuX
a=ssrc:4108384921 msid:- 100
a=ssrc:4108384921 mslabel:-
a=ssrc:4108384921 label:100
Additional context
Related links:
User reported this client error:
code -1: org.kurento.client.internal.server.KurentoServerException:Connection with server have been closed
and this server error:
[nioEventLoopGroup-3-1] org.kurento.jsonrpc.client.JsonRpcClientNettyWebSocket - channel closed
[AbstractJsonRpcClientWebSocket-disconnectExec-e4-t0] org.kurento.jsonrpc.client.AbstractJsonRpcClientWebSocket - Exception trying to reconnect to server ws://localhost:8888/kurento. Notifying disconnection
org.kurento.jsonrpc.JsonRpcException: Exception connecting to WebSocket server ws://localhost:8888/kurento
openvidu-server_1 | at org.kurento.jsonrpc.client.AbstractJsonRpcClientWebSocket.internalConnectIfNecessary(AbstractJsonRpcClientWebSocket.java:735)
openvidu-server_1 | at org.kurento.jsonrpc.client.AbstractJsonRpcClientWebSocket.connectIfNecessary(AbstractJsonRpcClientWebSocket.java:847)
openvidu-server_1 | at org.kurento.jsonrpc.client.AbstractJsonRpcClientWebSocket$16.run(AbstractJsonRpcClientWebSocket.java:635)
openvidu-server_1 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
openvidu-server_1 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
openvidu-server_1 | at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
openvidu-server_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
openvidu-server_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
openvidu-server_1 | at java.base/java.lang.Thread.run(Thread.java:829)
openvidu-server_1 | Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8888
openvidu-server_1 | Caused by: java.net.ConnectException: Connection refused
so I'd bet the Kurento Media Server is crashing catastrophically and OpenVidu just loses connectivity with it. I'll look into it.
Some updates:
Given the problematic SDP, where this is the important line:
m=video 9 UDP/TLS/RTP/SAVPF 0
this is already a malformed SDP because it requests static Payload Type 0, which is well defined and corresponds to PCMU Audio (see here: https://en.wikipedia.org/wiki/RTP_payload_formats)
So first of all, the client implementation is already buggy because it is requesting an audio format for a video media.
On top of that, Kurento Media Server is replying with a malformed Answer:
m=video 0 UDP/TLS/RTP/SAVPF
Where the port is set to 0 (which is good, it means "media rejected"), but it is malformed because the spec makes it mandatory to include some Payload Type (at least one) after the protocols. Any conforming client, when setting this as the remote SDP Answer, will error out. For example Chrome complains with this:
OperationError: Failed to execute 'setRemoteDescription' on 'RTCPeerConnection': Failed to parse SessionDescription. m=video 0 UDP/TLS/RTP/SAVPF Expects at least 4 fields.
A better Answer from Kurento would be like this:
m=video 0 UDP/TLS/RTP/SAVPF 0
where the original Payload Type is kept untouched. This would be the correct way for it to reject an invalid or malformed media line.
I've fixed this behavior in Kurento with Kurento/bugtracker#627
However we should still check why an error on the pc.setRemoteDescription() of the client would manage to bring a whole session down in OpenVidu. Or maybe there is some other thing that is also affected by this malformed SDP that was being returned by Kurento.
Also to check: why was OpenVidu losing connection to Kurento? In my tests, the Kurento server didn't crash or anything like that; it just returned a malformed SDP where the 4th field was missing, that's all.
Thank you for quickly responding to the issue and investigating it.
One thing I forgot to mention previously: we are using Mediasoup as the backend, not Kurento.
In our test case, we had two browser clients streaming video successfully, and when the faulty Android client joined the same session, the video streams of the browser clients froze immediately, every time (only the local video stream coming directly from the same device's camera stayed alive).
And despite of closing the session, it did not disappear from the session list. In server logs, it was marked as a ghost session. Each similar test produced a new ghost session, so we quickly had dozens of them.
OK I've been surgeon-testing the code this morning and found that in case of mediasoup, the issue is that the SDP Offer fails to be processed because there are no a=rtpmap lines describing the video codec(s). Thus mediasoup complains with an empty codecs error.
This error propagates up to the RPC protocol and reaches the application server (OpenVidu) so we still need to have a closer look at the server's logic to avoid breaking the session when this happens.
Side note: when using one of the standard Static PayloadType formats for RTP, there shouldn't be any need for a=rtpmap lines. The fact that mediasoup needs them means that it is not compatible with the mentioned static payloads. Which is curious but not surprising, given the almost exclusive focus that mediasoup has on WebRTC comms (and not general RTP comms). Still, interesting and a thing to note for possible future integrations with generic RTP endpoints.
After replicating the scenario with both Kurento and mediasoup as media server, I cannot get a session crash.
I have forcibly set up the SDP above in a session were two users were already successfully sending and receiving media. The only thing that happens is that the publish operation fails, with an error that properly propagates to the client side (and the catch block of the publish operation runs). The other two users were indifferent, and kept sending and receiving media without a problem. Evicting the three users from the session was enough to close the session, so no ghost session.
Here's some more details, maybe these will help in reproducing the issue:
- We are running OpenVidu Enterprise 2.20.0 (beta)
- Android phone with problems: Ulephone Armor 11T 5G
- Android client app: OpenVidu's official Android client example before my proposed fix was merged, for example the master branch in our fork: https://github.com/FinweLtd/openvidu-tutorials/tree/master/openvidu-android
Test case 1 (No problem):
- Open OpenVidu installation's phone app in browser (Macbook/Chrome) and create a new session, say renewed-black-beaver, with webcam enabled
- Open OpenVidu installation's phone app using another device (e.g. Galaxy S8 phone/Chrome), join the same session renewed-black-beaver, with webcam enabled
[Video flowing between the two clients] - Using official Android client as-is, except in strings.xml put proper default_openvidu_url and default_openvidu_secret, also default_session_name = renewed-black-beaver, build & run, then click JOIN from Ulephone's screen when the app starts
[Video flowing between all three clients] - Leave session from all three clients -> session is closed and disappears
Test case 2 (session crash):
-
Open OpenVidu installation's phone app in browser (Macbook/Chrome) and create a new session, say renewed-black-beaver, with webcam enabled
-
Open OpenVidu installation's phone app using another device (e.g. Galaxy S8 phone/Chrome), join the same session renewed-black-beaver, with webcam enabled
[Video flowing between the two clients] -
Modify Android client's code to use HW encoder, for example in Session.java's constructor:
final VideoEncoderFactory encoderFactory; final VideoDecoderFactory decoderFactory; //encoderFactory = new SoftwareVideoEncoderFactory(); //decoderFactory = new SoftwareVideoDecoderFactory(); EglBase eGlBase = EglBase.create(); encoderFactory = new HardwareVideoEncoderFactory( eGlBase.getEglBaseContext(), true, true); decoderFactory = new SoftwareVideoDecoderFactory(); -
build & run, then click JOIN from Ulephone's screen when the app starts
[Ulephone joins the session without video, video stops flowing between the two earlier clients, errors in logs] -
Leave session from all three clients -> session stays as a "ghost"
If necessary, we can also try to join your OpenVidu installation, to try to generate the issue "live" on your setup.
Just a warning: OpenVidu Enterprise beta 2.20.0 (mediasoup) will be discontinued in the following days. Your cluster will irretrievably shut down. We have warned all users with this version several times over the past two months. Please, update it ASAP.
Thanks, we have received the warning messages and installation of 2.22.0 to a new cluster is currently in progress.