Window event 'audioinput' not being triggered in Ionic 3

[Gabriel Freire]

Hi Guys, i'm very happy i found your plugin, i've been looking for a plugin with similar API to the web audio API for a long time, but i can't make the audioinput event get triggered

I'm emulating IOS/Android in the browser

The plugin was installed correctly.
I used this line on top of my Angular provider to make audioinput work

declare let audioinput: any;

My initialize function: the callback contains permission stuff and a method that calls audioinput.start() all permissions work perfectly

let captureCfg = {
      sampleRate: 16000,
      bufferSize: PROCESSING_BUFFER_LENGTH,
      channels: 1,
      format: audioinput.FORMAT.PCM_16BIT,
      audioSourceType: audioinput.AUDIOSOURCE_TYPE.DEFAULT,
      audioContext: this.audioContext
};
 audioinput.initialize(captureCfg, () => {...})

But when i call my start function , i'm using streamToWebAudio: true because i want to receive the data while recording in order to process, the event is never triggered.

audioinput.start({streamToWebAudio : true});
window.addEventListener('audioinput', (event) => {
    // THIS EVENT NEVER GETS TRIGGERED
     this.onAudioProcess(event);
}, false);
audioinput.connect(audioinput.getAudioContext().destination);

I did some debugging inside the plugin and the exec() function gets called with all the right properties but audioinput._audioInputEvent() never gets called, EVER.

What am i doing wrong? help me

[edin]

Hi, @gabrielfreire and thanks for your thorough issue description. I believe that that this is not a bug, but expected behaviour.

I believe that it is currently not clear from the documentation, since it is just mentioned in the comments in the basic usage example, so I will add a better explanation in the next update:

// Start with default values and let the plugin handle conversion of 
// raw data, and therefore will not send any audioinput events.

It would of course technically be possible to send events as well as forwarding the audio to web audio, but when I created this plugin, I didn't see the need for this, so it is currently only possible to process the audio using one of the supported methods, that is by using events or web audio.

So, when setting the streamToWebAudio parameter to true, the audioinput events are disabled per design.

I would recommend you to use a ScriptProcessor or later on the newer AudioWorklet (https://developers.google.com/web/updates/2017/12/audio-worklet) for getting access to the raw data from the audio node that you connect the audioinput plugin to.

You could also use the events instead (streamToWebAudio: false) to get the audio data, and then forward it to a web audio node in the same way the plugin does. This is done by the internal _getNextAudio and _getNextToPlay functions if you want to see how it is done.

Or you could create your own custom version of this plugin and modify the www/audioInputCapture.js code to always send audioinput events even if streamToWebAudio is enabled. Look for the audioinput._cfg.streamToWebAudio to find the proper places in the code to do this.

I hope this helps!

[Gabriel Freire]

Hi @edimuj, Thanks for the great explanation, this was very helpful.
I managed to make the audioinput event be triggered but it only works on a device or emulator.
It must be something specific to the device's microphone which the emulator mimics well, but not the browser.

I will try to connect audioinput to my ScriptProcessor node and see if it works. but before i try, would it look something like this ?
Creation of ScriptProcessor:

scriptProcessorNode = audioContext.createScriptProcessor(SOME_BUFFER_LENGTH, 1, 1);
scriptProcessorNode.onaudioprocess = (processingEvent) => {
    // some processing function
    onAudioProcess(processingEvent);
}

Connect audioinput to ScriptProcessor:

audioinput.connect(scriptProcessorNode);
scriptProcessorNode.connect(audioContext.destination);

I don't know if the scriptProcessor can receive the output from the device, does it come in the same format (AudioBuffer) like in processingEvent.inputBuffer: AudioBuffer ?

I'm still a bit confused about mixing the Device's output with the Web audio API and connecting everything together.

Thanks for the great plugin and great help.

[edin]

Yes, when using streamToWebAudio: true, the audioinput plugin will convert the raw audio from the microphone to a web audio stream with an internal Web Audio Node. It is this Audio node that you connect to your own Audio Node (in this case a scriptProcessorNode) using the audioinput connect() function.

I only use the device and emulator for this, so I cannot answer that. The browser implementation, was kindly implemented by @robertfromont

[edin]

I`m closing this. Feel free to open an issue if it still is a problem.

[Bobisback]

@gabrielfreire I am trying to use this plugin with ionic 4, do you by chance have a short guide on how to install this and use it in ionic 3?

Thanks,
Bob

P.S the main thing I am trying to figure out is how to install and import all the things from audioinput to use it in typescript. I am getting a audio input not defined issue. I do not think I am installing it correctly. any ideas?

[Gabriel Freire]

Hi @Bobisback , after installing it you need to ensure it was added on config.xml
if not, add this line
<plugin name="cordova-plugin-audioinput" spec="^1.0.1" />
change the spec value to your version
in typescript you need to declare global variables on the top of your files, like audioinput

declare let audioinput: any;

and then just make it listen for the event

window.addEventListener('audioinput', (event) => {
      /* processing event.data */;
}, false);

this is where i ask for the mic to record

                let captureCfg = {
                        sampleRate: 16000,
                        bufferSize: PROCESSING_BUFFER_LENGTH,
                        channels: 1,
                        format: audioinput.FORMAT.PCM_16BIT,
                        audioSourceType: audioinput.AUDIOSOURCE_TYPE.DEFAULT
                    };
                    audioinput.initialize(captureCfg, () => {
                        audioinput.checkMicrophonePermission((hasPermission) => {
                            if(hasPermission){
                                console.warn('Already have permission to record');
                                // startRecording
                                this.connectNodes();
                            } else {
                                console.warn('No permission to record yet');
                                console.warn('Asking...');
                                audioinput.getMicrophonePermission((hasPermission, message) => {
                                    if(hasPermission) {
                                        console.warn('User granted permission to record');
                                        this.connectNodes();
                                    } else {
                                        console.warn('User denied permission to record');
                                        this.status = RecordStatus.GETUSERMEDIA_ERROR;
                                    }
                                });
                            }
                        });
                    });

this is where i connect the nodes

audioinput.start({ bufferSize: PROCESSING_BUFFER_LENGTH, streamToWebAudio: false });
audioinput.connect(this.audioContext.destination);

p.s. above are my options, go to the docs to see what they mean

[Gabriel Freire]

Also make sure you have this on your package.json

"cordova": {
    "plugins": {
      "cordova-plugin-audioinput": {}
    }
}