MTHKView with many VideoEffects - preview is lagging
AdamAexol opened this issue · comments
Describe the bug
MTHKView with a couple of VideoEffects camera preview is lagging.
To Reproduce
I have MTHKView with camera preview and a couple of VideoEffects as camera overlays. Camera preview on MTHKView is lagging like this: https://youtu.be/nODP8NVQmN8
I identified that this commit is causing it:
d9ce742
Expected behavior
On the forked repository, I tried to fix it.
In file IOVideoUnit.swift with those lines:
var context: CIContext {
get {
return lockQueue.sync { self.videoMixer.context }
}
set {
lockQueue.async {
self.videoMixer.context = newValue
}
}
}
changed to this (lockQueue removed):
var context: CIContext {
get {
self.videoMixer.context
}
set {
self.videoMixer.context = newValue
}
}
camera preview started working fine https://youtu.be/65SxuXcXhyk
but on the other hand, sometimes there is a problem connecting to live stream. I guess calling those lines of code on lockQueue had a purpose.
The interesting thing is that it doesn't occur on every device.
On iPhone 7 Plus iOS 15.8, iPhone 12 iOS 17 camera preview is lagging
On iPhone 14 Pro iOS 17 everything is working smoothly
Do you have any ideas on how to fix this issue?
Version
1.7.1
Smartphone info.
iPhone 7 Plus iOS 15.8, iPhone 12 iOS 17 - those devices are affected
iPhone 14 Pro iOS 17 - works fine
Additional context
No response
Screenshots
No response
Relevant log output
No response
The issue could not be reproduced with the combination of the example app and iPhone 7 Plus. I would like to have sample code demonstrating how you are using the library.
I checked the fix above along with 1.7.2 release - preview lag is gone, but I can't connect to the live stream at all in this version.
I see something like this in the logs and it goes indefinitely:
2023-11-12 10:04:57.066 [Info] [com.haishinkit.HaishinKit] [VideoCodecSettings.swift:156] apply(_:rhs:) > bitRate change from 640000 to 8388608
2023-11-12 10:04:57.763 [Info] [com.haishinkit.HaishinKit] [IOAudioResampler.swift:227] setUp(_:) > inputFormat:<AVAudioFormat 0x281b3dd60: 1 ch, 48000 Hz, Int16>,outputFormat:<AVAudioFormat 0x281b3de00: 1 ch, 48000 Hz, Int16>
2023-11-12 10:04:57.764 [Info] [com.haishinkit.HaishinKit] [AudioCodec.swift:149] makeAudioConverter() > inputFormat:<AVAudioFormat 0x281b3de00: 1 ch, 48000 Hz, Int16>,outputFormat:<AVAudioFormat 0x281b3eda0: 1 ch, 48000 Hz, 'aac ' (0x00000002) 0 bits/channel, 0 bytes/packet, 1024 frames/packet, 0 bytes/frame>
2023-11-12 10:05:04.756 [Info] [com.haishinkit.HaishinKit] [RTMPNWSocket.swift:168] stateDidChange(to:) > Connection is ready.
2023-11-12 10:05:04.763 [Info] [com.haishinkit.HaishinKit] [RTMPNWSocket.swift:159] viabilityDidChange(to:) > Connection viability changed to true
2023-11-12 10:05:20.134 [Warn] [com.haishinkit.HaishinKit] [RTMPNWSocket.swift:179] stateDidChange(to:) > Connection failed:POSIXErrorCode(rawValue: 54): Connection reset by peer
2023-11-12 10:05:20.139 [Warn] [com.haishinkit.HaishinKit] [RTMPConnection.swift:405] on(status:) >
2023-11-12 10:05:22.166 [Info] [com.haishinkit.HaishinKit] [RTMPNWSocket.swift:168] stateDidChange(to:) > Connection is ready.
2023-11-12 10:05:22.170 [Info] [com.haishinkit.HaishinKit] [RTMPNWSocket.swift:159] viabilityDidChange(to:) > Connection viability changed to true
2023-11-12 10:05:37.635 [Warn] [com.haishinkit.HaishinKit] [RTMPConnection.swift:405] on(status:) >
2023-11-12 10:05:39.655 [Info] [com.haishinkit.HaishinKit] [RTMPNWSocket.swift:168] stateDidChange(to:) > Connection is ready.
2023-11-12 10:05:39.660 [Info] [com.haishinkit.HaishinKit] [RTMPNWSocket.swift:159] viabilityDidChange(to:) > Connection viability changed to true
2023-11-12 10:05:55.039 [Warn] [com.haishinkit.HaishinKit] [RTMPNWSocket.swift:179] stateDidChange(to:) > Connection failed:POSIXErrorCode(rawValue: 54): Connection reset by peer
2023-11-12 10:05:55.039 [Warn] [com.haishinkit.HaishinKit] [RTMPConnection.swift:405] on(status:) >
2023-11-12 10:05:57.071 [Info] [com.haishinkit.HaishinKit] [RTMPNWSocket.swift:168] stateDidChange(to:) > Connection is ready.
2023-11-12 10:05:57.078 [Info] [com.haishinkit.HaishinKit] [RTMPNWSocket.swift:159] viabilityDidChange(to:) > Connection viability changed to true
2023-11-12 10:06:12.489 [Warn] [com.haishinkit.HaishinKit] [RTMPNWSocket.swift:179] stateDidChange(to:) > Connection failed:POSIXErrorCode(rawValue: 54): Connection reset by peer
2023-11-12 10:06:12.490 [Warn] [com.haishinkit.HaishinKit] [RTMPConnection.swift:405] on(status:) >
2023-11-12 10:06:14.516 [Info] [com.haishinkit.HaishinKit] [RTMPNWSocket.swift:168] stateDidChange(to:) > Connection is ready.
2023-11-12 10:06:14.524 [Info] [com.haishinkit.HaishinKit] [RTMPNWSocket.swift:159] viabilityDidChange(to:) > Connection viability changed to true
2023-11-12 10:06:30.012 [Warn] [com.haishinkit.HaishinKit] [RTMPConnection.swift:405] on(status:) >
2023-11-12 10:06:32.046 [Info] [com.haishinkit.HaishinKit] [RTMPNWSocket.swift:168] stateDidChange(to:) > Connection is ready.
2023-11-12 10:06:32.051 [Info] [com.haishinkit.HaishinKit] [RTMPNWSocket.swift:159] viabilityDidChange(to:) > Connection viability changed to true
2023-11-12 10:06:47.407 [Warn] [com.haishinkit.HaishinKit] [RTMPNWSocket.swift:179] stateDidChange(to:) > Connection failed:POSIXErrorCode(rawValue: 54): Connection reset by peer
Regarding the connection issue, I'm not quite sure because it works on my end. I would like you to review if there are any issues with the network or other related aspects.
This issue itself has been completed, so I will close it.