[RTMPNWSocket] The video bandwidth is insufficient upon reconnection.

Question

[RTMPNWSocket] The video bandwidth is insufficient upon reconnection.

shogo4405 opened this issue a year ago · comments

Describe the bug

An issue I have is after reconnection the way publishSufficientBWOccured is quite different:
It's getting sufficient but currentBytesOut are consistently lower,
For example, setting bitrate to 2765000 (2.7 mbps) consistently pushing out ~501216 bits per second (500kbps)
next second it still pushes only 500kbps or so even with higher bitrate

To Reproduce

In Sample app (SwiftUI iOS)
Start with Wifi + 5G
Swipe to controls, turn off WiFi
I'm getting these log messages:

Expected behavior

N/A

Version

N/A

Smartphone info.

No response

Additional context

No response

Screenshots

No response

Relevant log output

No response

Mark Sergienko · Answer 1 · Mon Jun 05 2023 06:22:37 GMT+0800 (China Standard Time)

I tested it some more, and it definitely is hard to impossible to re-establish connection while on LTE/5G now.. Or the connection is much slower once re-established, even if initially it was good speed (same location).
It seems to re-connect well with good speed on WiFi.

iPhone 13 Pro
Latest HaishinKit (main)
iOS 16.1
Vodafone Portugal

shogo4405 · Answer 2 · Mon Jun 05 2023 23:26:02 GMT+0800 (China Standard Time)

It seems unrelated to this. I can't reproduce it.
#1203 (comment)

iOS 16.5 docomo Japan

SETTINGS

        rtmpStream.videoSettings = VideoCodecSettings(
            videoSize: .init(width: 1280, height: 720),
            profileLevel: kVTProfileLevel_H264_Baseline_AutoLevel as String,
            bitRate: 2500 * 1000,
            maxKeyFrameIntervalDuration: 2,
            scalingMode: .trim,
            bitRateMode: .average,
            allowFrameReordering: nil,
            isHardwareEncoderEnabled: true
        )

LTE ON

2023-06-06 00:01:37.996 [Info] [com.haishinkit.Exsample.iOS] [LiveViewController.swift:210] rtmpStatusHandler(_:) > NetConnection.Connect.Success
2023-06-06 00:01:38.102 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>3420
2023-06-06 00:01:38.520 [Info] [com.haishinkit.Exsample.iOS] [LiveViewController.swift:210] rtmpStatusHandler(_:) > NetStream.Publish.Start
2023-06-06 00:01:39.102 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>82996
2023-06-06 00:01:40.101 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>76203
2023-06-06 00:01:41.101 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>183553
2023-06-06 00:01:42.101 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>131605
2023-06-06 00:01:43.101 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>89246
2023-06-06 00:01:44.101 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>246938
2023-06-06 00:01:45.101 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>218990

LTE OFF -> ON

00:01:52.513 [Info] [com.haishinkit.Exsample.iOS] [LiveViewController.swift:210] rtmpStatusHandler(_:) > NetConnection.Connect.Success
2023-06-06 00:01:52.663 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>3420
2023-06-06 00:01:52.993 [Info] [com.haishinkit.Exsample.iOS] [LiveViewController.swift:210] rtmpStatusHandler(_:) > NetStream.Publish.Start
2023-06-06 00:01:53.663 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>67867
2023-06-06 00:01:54.664 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>79618
2023-06-06 00:01:55.663 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>178283
2023-06-06 00:01:56.663 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>118400
2023-06-06 00:01:57.663 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>171120
2023-06-06 00:01:58.663 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>132234
2023-06-06 00:01:59.663 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>179937
2023-06-06 00:02:00.663 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>134678
2023-06-06 00:02:01.663 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>174380

LTE OFF -> ON

2023-06-06 00:02:09.470 [Info] [com.haishinkit.Exsample.iOS] [LiveViewController.swift:210] rtmpStatusHandler(_:) > NetConnection.Connect.Closed
2023-06-06 00:02:11.479 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>0
2023-06-06 00:02:12.477 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>0
2023-06-06 00:02:13.164 [Info] [com.haishinkit.Exsample.iOS] [LiveViewController.swift:210] rtmpStatusHandler(_:) > NetConnection.Connect.Success
2023-06-06 00:02:13.477 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>3547
2023-06-06 00:02:13.633 [Info] [com.haishinkit.Exsample.iOS] [LiveViewController.swift:210] rtmpStatusHandler(_:) > NetStream.Publish.Start
2023-06-06 00:02:14.477 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>83654
2023-06-06 00:02:15.477 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>90796
2023-06-06 00:02:16.477 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>172142
2023-06-06 00:02:17.476 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>125535
2023-06-06 00:02:18.477 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>173675
2023-06-06 00:02:19.477 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>131680
2023-06-06 00:02:20.477 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>204814
2023-06-06 00:02:21.477 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:462] on(timer:) > currentBytesOutPerSecond=>154720

Mark Sergienko · Answer 3 · Tue Jun 06 2023 23:47:26 GMT+0800 (China Standard Time)

Ok, I've reset bitrate to default before reconnect and it may have helped..
But there's one problem: after some reconnects if NetConnection.Connect.Failed then on next reconnect attempt my rtmpConnection.connect() hangs for a very very long time.. with no indication of what's happening, no timeout too ...

Mark Sergienko · Answer 4 · Wed Jun 07 2023 19:46:59 GMT+0800 (China Standard Time)

Ok, here's the log where I have the reconnection stuck forever, but I can't yet reproduce it in your Example app.. So maybe it's something very rare, but can anything be done with it to timeout the reconnection?


2023-06-06 19:40:06.702179+0100 appnew[14115:2955040] Target bitrate 2764800
2023-06-06 19:40:07.593487+0100 appnew[14115:2955995] [] nw_flow_prepare_output_frames Failing the write requests [57: Socket is not connected]
2023-06-06 19:40:07.593587+0100 appnew[14115:2955995] [connection] nw_write_request_report [C25] Send failed with error "Socket is not connected"
2023-06-06 19:40:07.594403+0100 appnew[14115:2955995] [] nw_flow_prepare_output_frames Failing the write requests [57: Socket is not connected]
2023-06-06 19:40:07.594486+0100 appnew[14115:2955995] [connection] nw_write_request_report [C25] Send failed with error "Socket is not connected"
2023-06-06 19:40:07.594801+0100 appnew[14115:2955995] [] nw_flow_prepare_output_frames Failing the write requests [57: Socket is not connected]
2023-06-06 19:40:07.594855+0100 appnew[14115:2955995] [connection] nw_write_request_report [C25] Send failed with error "Socket is not connected"
2023-06-06 19:40:07.595 [Warn] [com.haishinkit.HaishinKit] [RTMPConnection.swift:406] on(status:) > 
NetConnection.Connect.Closed
Reason: connectFailed/connectClosed
waiting for new connect attempt ... 
status connecting
2023-06-06 19:40:07.596629+0100 appnew[14115:2957421] [connection] nw_read_request_report [C25] Receive failed with error "Socket is not connected"
New connect attempt ...  0
NetConnection.Connect.Failed
Reason: connectFailed/connectClosed
waiting for new connect attempt ... 
status connecting
sufficient 2764800 0
New connect attempt ...  1
sufficient 2764800 0
NetConnection.Connect.Failed
Reason: connectFailed/connectClosed
waiting for new connect attempt ... 
status connecting
sufficient 2764800 0
sufficient 2764800 0
sufficient 2764800 0
sufficient 2764800 0
New connect attempt ...  2
sufficient 2764800 0
sufficient 2764800 0
sufficient 2764800 0
sufficient 2764800 0
sufficient 2764800 0
sufficient 2764800 0
sufficient 2764800 0
sufficient 2764800 0
sufficient 2764800 0

I'm using LTE only, disconnecting, connecting, sometimes with small pause, sometimes quickly

shogo4405 · Answer 5 · Wed Jun 07 2023 20:24:15 GMT+0800 (China Standard Time)

The connection default timeout is 15 seconds. Additionally, it can be handled in .ioError event.

rtmpConneciton.timeout = 15 // sec
rtmpConnection.addEventListener(.ioError, selector: #selector(rtmpErrorHandler), observer: self)

Mark Sergienko · Answer 6 · Thu Jun 15 2023 17:18:21 GMT+0800 (China Standard Time)

@shogo4405 I was able to reproduce the issue with the SwiftUI Example app.
Strangely it depends on the bitrate setting:

How I'm testing:

Turn WiFi off
Connect on 5G
Swipe to control center, disconnect Cellular
Connect Cellular
Connection restores

Reconnection works every time in test app until I change bitrate setting:

Steps to reproduce:

ViewModel:

// @Published var videoRate = CGFloat(VideoCodecSettings.default.bitRate / 1000)
@Published var videoRate = CGFloat(2700);

config () {  
 ...
        rtmpStream.videoSettings.videoSize = .init(width: 720, height: 1280)
        rtmpStream.videoSettings.bitRate = UInt32(videoRate*1000)

Views:

Slider(
                            value: $viewModel.videoRate,
                            in: 32...4000,
...
)

Reconnection doesn't happen

shogo4405 · Answer 7 · Mon Jun 19 2023 21:13:08 GMT+0800 (China Standard Time)

Also I can't reproduce it 🧐 with the SwiftUI Example app. I tested with YouTube. It's difficult to determine whether the issue with the carrier's network or the server. Personally, I have a good network environment.

main...feature/bitrate-tests

Camera with moving.

2023-19-06 21:57:31.316 [Info] [com.haishinkit.HaishinKit.iOSSwiftUI] [ViewModel.swift:135] startPublish() > rtmp://a.rtmp.youtube.com/live2
NetConnection.Connect.Success
2023-19-06 21:57:32.320 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️3420
NetStream.Publish.Start
2023-19-06 21:57:33.318 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️196676
2023-19-06 21:57:34.318 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️102404
2023-19-06 21:57:35.318 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️299819
2023-19-06 21:57:36.318 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️325817
2023-19-06 21:57:37.318 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️359717
2023-19-06 21:57:38.318 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️334060
2023-19-06 21:57:39.318 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️364704
2023-19-06 21:57:40.318 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️373279
2023-06-19 21:57:40.938553+0900 Example iOS+SwiftUI[4996:2037831] [] nw_flow_prepare_output_frames Failing the write requests [57: Socket is not connected]
2023-06-19 21:57:40.938655+0900 Example iOS+SwiftUI[4996:2037831] [connection] nw_write_request_report [C1] Send failed with error "Socket is not connected"
2023-19-06 21:57:40.942 [Warn] [com.haishinkit.HaishinKit] [RTMPConnection.swift:406] on(status:) > 
NetConnection.Connect.Closed
2023-06-19 21:57:40.942550+0900 Example iOS+SwiftUI[4996:2037831] [connection] nw_read_request_report [C1] Receive failed with error "Socket is not connected"
2023-19-06 21:57:42.949 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️0
2023-19-06 21:57:43.949 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️0
NetConnection.Connect.Success
2023-19-06 21:57:44.948 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️3547
NetStream.Publish.Start
2023-19-06 21:57:45.949 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️220965
2023-19-06 21:57:46.949 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️113383
2023-19-06 21:57:47.948 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️276863
2023-19-06 21:57:48.952 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️346059
2023-19-06 21:57:49.953 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️375399
2023-19-06 21:57:50.949 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️551337
2023-19-06 21:57:51.949 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️403280
2023-19-06 21:57:52.949 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️355728
2023-19-06 21:57:53.948 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️373155

Camera is stationary

2023-19-06 22:09:58.885 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️3547
NetStream.Publish.Start
2023-19-06 22:09:59.885 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️242489
2023-19-06 22:10:00.885 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️80572
2023-19-06 22:10:01.885 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️328537
2023-19-06 22:10:02.885 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️199453
2023-19-06 22:10:03.885 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️447905
2023-19-06 22:10:04.884 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️320155
2023-19-06 22:10:05.884 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️441931
2023-06-19 22:10:06.243352+0900 Example iOS+SwiftUI[5003:2047899] [] nw_flow_prepare_output_frames Failing the write requests [57: Socket is not connected]
2023-06-19 22:10:06.243475+0900 Example iOS+SwiftUI[5003:2047899] [connection] nw_write_request_report [C1] Send failed with error "Socket is not connected"
2023-19-06 22:10:06.245 [Warn] [com.haishinkit.HaishinKit] [RTMPConnection.swift:406] on(status:) > 
2023-06-19 22:10:06.244937+0900 Example iOS+SwiftUI[5003:2047895] [connection] nw_read_request_report [C1] Receive failed with error "Socket is not connected"
NetConnection.Connect.Closed
2023-19-06 22:10:08.249 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️0
2023-19-06 22:10:09.252 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️0
NetConnection.Connect.Success
2023-19-06 22:10:10.249 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️3547
NetStream.Publish.Start
2023-19-06 22:10:11.249 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️234860
2023-19-06 22:10:12.249 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️115643
2023-19-06 22:10:13.248 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️347205
2023-19-06 22:10:14.249 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️167122
2023-19-06 22:10:15.249 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️456451
2023-19-06 22:10:16.249 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️146516
2023-19-06 22:10:17.249 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️433638
2023-19-06 22:10:18.249 [Info] [com.haishinkit.HaishinKit] [RTMPConnection.swift:458] on(timer:) > ⭐️175761

Mark Sergienko · Answer 8 · Thu Jun 22 2023 00:40:22 GMT+0800 (China Standard Time)

I reproduced a state where the example app does not see that it's disconnected, and keeps showing 30 FPS:

Start on 5G
Go to control center. Turn on WiFi, turn off 5G

Go to control center. Turn on 5G, turn off WiFi

Go to control center. Turn on WiFi, turn off 5G =>

The app is showing 30FPS but it's really not connected (Race condition?)

Also sometimes getting
2023-06-21 17:38:31.481493+0100 Example iOS+SwiftUI[5310:1041968] [tcp] tcp_input [C12.1.1:2] flags=[R] seq=1314460444, ack=0, win=0 state=CLOSE_WAIT rcv_nxt=1314460469, snd_una=2821809103

But connection private func stateDidChange(to state: NWConnection.State) { Does not receive the state change..

Mark Sergienko · Answer 9 · Thu Jun 22 2023 20:26:48 GMT+0800 (China Standard Time)

I dive deeper and found 2 distinct issues in RTMPNWSocket, and working on a pull request:

Sometimes connection is not viable, but the socket is trying to send a final message with completion that never happens and so it hangs.
Sometimes connection is in CLOSE_WAIT forever without updating the NWConnection.State (I guess because it wasn't closed by connection?.cancel(), working to find out more. Meanwhile RTMPConnection thinks it's active but pushing 0 bytes out, and does not allow to reconnect.

shogo4405 · Answer 10 · Mon Jul 03 2023 19:23:07 GMT+0800 (China Standard Time)

It seems that there is a hang in the outputQueue. I have successfully addressed this issue on my end with commit 25daa56.