Recording audio using AudioKit while playing an audio file produces very loud feedback loop
Samigos opened this issue · comments
macOS Version(s) Used to Build
macOS 13.5 Ventura
Xcode Version(s)
Xcode 15
AudioKit Version
5.6.1
Description
I'm trying to record audio and apply real-time pitch shifting and write the output to an AVAudioFile, all that using AudioKit. At the same time, I want to play an audio file for the duration of the recording, using AVPlayer.
The thing is that if I play the audio file during the recording, there's a very high-pitch feedback loop echoing very loudly! If on the other hand I perform the recording simply by using AVAudioRecorder, there's zero feedback loop.
Here's how I record the processed and unprocessed audio using taps on AudioKit's engine; (you'll notice that I set both AVAudioSession's and AudioKit's Settings category.)
class PitchCorrectionAudioKitService {
private let engine = AudioEngine()
private var pitchShiftEffect: PitchShifter!
private var unprocessedAudioFile: AVAudioFile?
private var processedAudioFile: AVAudioFile?
// ------------------
@Injected private var pitchDetectionService: PitchDetectionRepository
// ------------------
init() {
guard let input = engine.input else { return }
pitchShiftEffect = PitchShifter(input)
engine.output = pitchShiftEffect
}
func start(baseNote: Note) {
handleAudioSessionCategory()
pitchDetectionService.start { data in
self.autoCorrectPitch(data, baseNote: baseNote)
}
try! engine.start()
engine.input?.avAudioNode.installTap(onBus: 0, bufferSize: 4096, format: nil) { buffer, time in
try! self.unprocessedAudioFile?.write(from: buffer)
}
engine.output?.avAudioNode.installTap(onBus: 0, bufferSize: 4096, format: nil) { buffer, time in
try! self.processedAudioFile?.write(from: buffer)
}
}
private func handleAudioSessionCategory() {
if UIDevice.primaryAudioDevice == .defaultSpeakers {
try! AVAudioSession.sharedInstance().setCategory(.playAndRecord, options: [.defaultToSpeaker, .allowBluetooth])
try! Settings.setSession(category: .playAndRecord, with: [.defaultToSpeaker, .allowBluetooth])
} else {
try! AVAudioSession.sharedInstance().setCategory(.playAndRecord, mode: .videoRecording)
try! Settings.setSession(category: .playAndRecord)
}
try! AVAudioSession.sharedInstance().setActive(true)
}
}
The moment I start recording, I also play a music track;
private var musicTrackPlayer: AVPlayer?
musicTrackPlayer?.pause()
musicTrackPlayer?.volume = areHeadphonesConnected ? 0.4 : 0.01
musicTrackPlayer?.play()
Any ideas?
Crash Logs, Screenshots or Other Attachments (if applicable)
No response
Hi @Samigos
Thank you for your input! I will look into this and get back to you shortly.
In the meantime, here's something interesting I found: https://stackoverflow.com/questions/58472683/cancel-ios-microphone-echo-cancellation-and-noise-suppression.
--Evan
Hi @Samigos,
I've looked into this and verified feedback is indeed happening when the audio is being recorded with playback audio. Would you be able to change your code to use the measurement mode explained in the post above?
Using this mode while recording will remove any signal processing included Apple's AVAudioEngine by default which could cause the feedback (i.e. automatic gain settings). According to the AVAudioRecorder Documentation, these signal processing functions are only included with the engine and not the recorder itself which may explain why it doesn't feedback without the engine.
AudioKit also uses the shared instance by default, so you should be able to activate measurement mode with these lines only:
try! AVAudioSession.sharedInstance().setCategory(.playAndRecord, mode: .measurement, options: [.defaultToSpeaker, .allowBluetooth])
Then you can set it active with try! AVAudioSession.sharedInstance().setActive(true) as you did at the end of your handleAudioSessionCategory() function.
Please let me know if these steps meet the requirements of your implementation. If so, I'll add the mode handling to AudioKit's Settings session management along with the tests and documentation for this issue. Thanks.
--Evan
Hi @emurray2 and thank you for taking the time to help me!
Unfortunately, I can't use .measurement because it seems to mess up PitchTap. Is there anything else that comes to mind? I've been trying to make this work for days now and I'm starting to worry it might not be possible.
Also, what I don't understand is how is there any feedback, if the microphone doesn't produce any sound to the speaker?
@Samigos Anytime, happy to help and I understand. AVFoundation can be one of the trickiest API to work with in my opinion.
The thing is that if I play the audio file during the recording, there's a very high-pitch feedback loop echoing very loudly! If on the other hand I perform the recording simply by using AVAudioRecorder, there's zero feedback loop.
Could you please link your code which demonstrates use of AVAudioRecorder that doesn't create the feedback?
Also, what I don't understand is how is there any feedback, if the microphone doesn't produce any sound to the speaker?
pitchShiftEffect = PitchShifter(input)
engine.output = pitchShiftEffect
In these two lines, a pitch shifter is initialized which produces an output from the engine input (microphone). The output of pitch shifter is then set to the engine output (speaker). Whenever the input is in a path to the engine output, there is an increased probability of acoustic feedback occurring. This depends on many things, such as the input volume, output volume, and properties of any intermediate systems interacting with input/output (the pitch shifter in this case).
In our case, pitch shifter actually produces a pitch shifted version of the microphone input as its output and sends it directly to the speakers. This is why feedback is occurring because the microphone takes in the output from the speaker and sends it back through the pitch shifter again.
Could you please link your code which demonstrates use of
AVAudioRecorderthat doesn't create the feedback?
I didn't explain myself correctly, I apologise for that... There was no feedback loop when there was no involvement of AVAudioEngine/AudioKit, just AVAudioRecorder and AVPlayer. (simpler times I guess 😜)
In these two lines, a pitch shifter is initialized which produces an output from the engine input (microphone). The output of pitch shifter is then set to the engine output (speaker). Whenever the input is in a path to the engine output, there is an increased probability of acoustic feedback occurring. This depends on many things, such as the input volume, output volume, and properties of any intermediate systems interacting with input/output (the pitch shifter in this case).
In our case, pitch shifter actually produces a pitch shifted version of the microphone input as its output and sends it directly to the speakers. This is why feedback is occurring because the microphone takes in the output from the speaker and sends it back through the pitch shifter again.
Is there no way that the pitch shifter isn't connected to any audio output? All I want is to detect, shift accordingly and write the modified buffer, all in real-time. I'm not going to play it back.
I didn't explain myself correctly, I apologise for that... There was no feedback loop when there was no involvement of AVAudioEngine/AudioKit, just AVAudioRecorder and AVPlayer. (simpler times I guess 😜)
@Samigos No, your explanation matches my interpretation. I realize the audio plays fine without AVAudioEngine and AudioKit. What I'm asking is if you have an example of the code where there was no feedback (or the code that didn't use AudioKit). That would be helpful to figure out where the issue is originating from.
Is there no way that the pitch shifter isn't connected to any audio output? All I want is to detect, shift accordingly and write the modified buffer, all in real-time. I'm not going to play it back.
As I said above, the pitch shifter is connected to the audio output whenever you set it to the engine's output because this could be the speaker in the scenario you use .defaultToSpeaker. If you don't want to route pitch shifter to the output and not have any output, you should put an empty Mixer() at the engine output instead of pitch shifter. You can also use something like NodeRecorder instead of taps to record the output from pitch shifter and write it to a file or buffer. See: https://github.com/AudioKit/AudioKit/blob/main/Sources/AudioKit/Taps/NodeRecorder.swift
Then, if you wanted to at a later point, you can load the recorded file produced from NodeRecorder in an AudioPlayer and play the recorded audio just fine. We also have many examples of these use cases you can view in our Cookbook: https://github.com/AudioKit/Cookbook/blob/main/Cookbook/CookbookCommon/Sources/CookbookCommon/Recipes/MiniApps/Recorder.swift .
Hey there! I'm sorry it took me this long to answer, but I was caught up with work. Now that I'm finished, I can tell you how I managed to solve it.
If you don't want to route pitch shifter to the output and not have any output, you should put an empty Mixer() at the engine output instead of pitch shifter.
I couldn't make it work with an empty Mixer, there was a crash but I don't remember what exactly. Instead, I used a Fader with gain set to 0.
pitchShiftEffect = PitchShifter(input)
silencer = Fader(pitchShiftEffect, gain: 0)
mixer.addInput(silencer)
engine.output = mixer
That way, I could get the pitched buffers and write them to a .caf file.
pitchShiftEffect.avAudioNode.installTap(onBus: 0, bufferSize: 4096, format: nil) { [weak self] buffer, time in
try? self?.processedAudioFile?.write(from: buffer)
}
The next issue was the audio quality. After lots and lots of attempts, I ended up with this setup:
private func handleAudioSessionCategory() {
Settings.disableAVAudioSessionCategoryManagement = true
do {
try AVAudioSession.sharedInstance().setCategory(.playAndRecord, mode: .videoRecording, options: [.allowBluetooth])
try AVAudioSession.sharedInstance().setActive(true)
let areHeadphonesConnected = UIDevice.primaryAudioDevice != .defaultSpeakers
if areHeadphonesConnected {
try AVAudioSession.sharedInstance().setCategory(.playAndRecord, mode: .videoRecording)
} else {
try AVAudioSession.sharedInstance().setCategory(.playback)
}
try AVAudioSession.sharedInstance().setActive(true)
} catch {}
}
As far as you first question is concerned:
@Samigos No, your explanation matches my interpretation. I realize the audio plays fine without AVAudioEngine and AudioKit. What I'm asking is if you have an example of the code where there was no feedback (or the code that didn't use AudioKit). That would be helpful to figure out where the issue is originating from.
It was just a plain recorder, like this one:
func startRecordingAudio() {
let settings = [
AVFormatIDKey: Int(kAudioFormatMPEG4AAC),
AVSampleRateKey: 44100,
AVNumberOfChannelsKey: 2,
AVEncoderAudioQualityKey: AVAudioQuality.high.rawValue
]
audioFileURL = URL(fileURLWithPath: NSTemporaryDirectory()).appendingPathComponent("audioRecorderTemp\(Date().timeIntervalSince1970)").appendingPathExtension("m4a")
do {
audioRecorder = try AVAudioRecorder(url: audioFileURL!, settings: settings)
audioRecorder?.record()
} catch {
finishRecordingAudio(success: false)
}
}
func finishRecordingAudio(success: Bool) {
audioRecorder?.stop()
audioRecorder = nil
}
That's all from me! 😁
Thank you very much for helping me!
@Samigos Anytime, glad you were able to get it working! To confirm--there's nothing we need to fix with AudioKit right? Or is it mostly AVAudioEngine? I wanted to double check in case there were any improvements you had in mind.
I don’t think so, no. It was just that I hadn’t used the library before, so I didn’t know how to use it. In my defence, it's a huuuuge library! 😁
I don’t think so, no. It was just that I hadn’t used the library before, so I didn’t know how to use it. In my defence, it's a huuuuge library! 😁
I'm with you there, it takes a while to learn since there's so much stuff. For future debugging and implementation help, I definitely recommend joining our AudioKit Discord server. There's so many smart people in there who can help make that learning curve a bit easier. If you want, I can send the invite link to your GitHub email. Let me know if that sounds good and I'll send you the link. For now, since this isn't an issue with AudioKit I'll go ahead and close it, but feel free to re-open if you find there's an issue in the implementation of AudioKit related to acoustic feedback.