Updating the position of Mixer3D node does not affect output volume
j-j-m opened this issue · comments
macOS Version(s) Used to Build
macOS 13 Ventura
Xcode Version(s)
Xcode 14
Description
after seeing #2830 I was super stoked to use AudioKit in my game but noticed a few issues
-
I was not able to dynamically add spatial mixers to the environment after the engine had been started. would be wonderful to see how this could be used in a 3d environment where objects can be dynamically created and added to a scene. if helpful I can create another ticket for this.
-
updating the position of the 3d mixer has no effect it seems.
to demonstrate this I have forked the demo project (https://github.com/j-j-m/AudioKit3DDemo)
and updated the sound source to be affected by physics
doing so necessitates referencing presentation.worldPosition for the updated position
updateAudioSourceNodeDelegate?.updateSoundSourcePosition(AVAudio3DPoint(
x: soundSource.presentation.worldPosition.x,
y: soundSource.presentation.worldPosition.y,
z: soundSource.presentation.worldPosition.z))for convenience I am printing the value before updating it
func updateSoundSourcePosition(_ position3D: AVAudio3DPoint) {
print(position3D)
source1mixer3D.position = position3D
}Crash Logs, Screenshots or Other Attachments (if applicable)
No response
Does the issue you described show up in the Cookbook example too? If so, maybe @ForeverTangent can take a look.
@j-j-m Hey Jacob.
For question 1, the AVAudioEngine (and by extension AudioKit) doesn't let you add Spatial Mixers after the engine has started. (Well you 'can' add it, but the AVAudioEngine just ignores it) My apologies... I forgot to add those minutia details to the documentation. (But in my defense it is one of those things buried in Apple's documentation I stumbled across once, I could not find again if my life depended on it.) If you want multiple node you have to set them up before you start the engine, or change the EnvironmentalNode settings on the fly. Or just stop the engine re-adjust the graph and start it up again.
As for 2 let me download your demo and take a look at it.
@j-j-m
OK First Gravity seemed to get turned on in your demo. So the Sound Source Ball was sinking into oblivion.
Turn that off, and you should be able to hear the spatial mixer update.
Second, you have to remember you are dealing with 2 3D points in the demo... the Position of the Sound Source (the green ball), and the Listener (the camera/ears).
If everything is working correctly... the sound source's position should NOT change. It is the camera/ears that is the flying around the world that is changing.
You need to poke
renderer.pointOfView the SceneKit Camera
or
'listenerPosition3D' (the Ears) to see what is actually changing.
Hope that helps.
hey @ForeverTangent @NickCulbertson thanks for looking at this
starting with part 2. you will notice that my code handles the world position of the object and so the falling into oblivion is by design... I wanted to demonstrate that the audio did not diminish even though the ball falls off the map ForeverTangent/AudioKit3DDemo@780fe86
As for part one I am able to to this with SCNAudioPlayer attached directly to an SCNNode.
I guess that gets at my use case.
My game consists of a character that the user can control, with a camera that follows the character. via SCNPhysicsContactDelegate I can when the characters feet intersect the ground. I use an SCNAudioPlayer to trigger a footstep sound
likewise there are other characters in the game, and they make sounds in a similar way. with SCNAudioPlayer I can spatialize this audio.
@ForeverTangent maybe my use case is not supported via AudioKit? I should expect to be able to use it with a static scene graph where the only thing that moves is the users point of view?
I have been able to get spatialization working with SCNAudioPlayer but really would like to have more control and include generative audio in my game. I have been able to create the various footstep sounds I need... but just can't get them working in a spatial audio context. I even tried initializing SCNAudioPlayer with an AVAudioNode exposed by Node but I couldn't get anything beyond simple oscillators to work. there's probably more to investigate there, but I haven't had time.
anyways. thanks again for looking at this.
Hey @j-j-m,
Thanks for the clarification. Ok I get what is going on now.
Yeah, you sort of hit the core of the problem with this.
@ForeverTangent maybe my use case is not supported via AudioKit? I should expect to be able to use it with a static scene graph where the only thing that moves is the users point of view?
The problem is SCNAudioPlayer is not part of AudioKit. It is SceneKit's own audio player for its own instance of its own AVAudioEngine. It would be a completely different instance and audio graph of the AVAudioEngine than what AudioKit is using, so that is why you haven't been able to get it working.
IE SCNAudioPlayer -> AudioKit
(Now let me put on my old game developer hat (I used to work in sound for Activision years ago)).
Now mind, I am just theorizing here but what you might be able to do is to go the other way.
AudioKit -> SCNAudioPlayer
It looks like the SCNAudioPlayer can be initialized with an AVAudioNode; and all AudioKit nodes are/contain an AVAudioNode.
So build you custom audio generator chain in AudioKit, but do not connect it to the AudioKit.AudioEngine. Instead, take the last element in the chain (probably best if it was a Mixer3D, not Mixer) and grab the AVAudioNode from that AudioKit Node. Then use that AVAudioNode to when initializing the SCNAudioPlayer.
You will probably have to build a custom protocol to trigger the sounds, from the SCNPhysicsContactDelegate, but in theory it seems to work (at least in my head).
Final last bit of wisdom from an old curmudgeon:
As for triggering the sounds of steps via physics engine. It is a noble idea, but you might be better off tying the sound to any animation of the character walking. The reason why, is many sound designers have gone down that same road (myself included) of wanting the physics engine to trigger the character walking sounds. It great until you have the experience the physic engine spamming the game with the steps of a million characters, or (even better) the one un-moving character step that never seems to end.
That might not happen this time, but I am warning you all the same. As they say in StarFox, "Good Luck!"