Implement plugin UI and support user params
benwiley4000 opened this issue · comments
We should expose params as a global samplePluginParams in a plugin. We can accept boolean or number params. Not sure if we should specify ranges or not.
The params should remain editable if a plugin unloads so they can be fixed before reloading.
We should also support sampleImportPlugin, which can return a list of audio buffers.
All sampleparam numbers are values between 0 and 1
Plugins can return either AudioBuffer or AudioNode. If the return value is an AudioNode we pipe it to a ScriptProcessorNode that records the output.
We also need to enforce for samplePlugin that the length of the output is the same as the input. Otherwise it gets complicated to track duration.
If the user changes the sample rate of the returned audio buffer for some reason (possible), we should read it into the audio context to get a resampling. We should test to make sure this works by the way. Otherwise use the wave-resampler library.
I decided not to enforce that the returned audiobuffer needs to have the same length. This would proclude that a plugin that could, for example, timestretch audio, which might be desired.
In order to make this all work, I need to remove the duration cached field, which can be computed at runtime, and replace it with a pluginDurationCoefficient which can be multiplied against the duration to get the untrimmed duration of the processed sample. Whenever this coefficient changes we also need to update the trim frames to a rounded new value as a function of the ratio between the old and new coefficients. One advantage of storing this ratio is that it makes it simpler to decide later to apply plugins post-trim, but to start we can apply plugins before any other processing.
Plugins will be javascript files and we can use their filename as the plugin name. We can uniquely identify plugins with a content hash.
We will need a plugin indexeddb store that includes ids (hashes), filenames, and file content. We can validate hashes against their content after lookup as needed. The plugins should be included in a subdirectory of the export zip.
To the sample metadata, besides the coefficient in the cachedData we will want to add a list of plugins. It can include filenames (unused for lookup but helpful for reading), id hashes, and plugin params for each plugin.
Finally perhaps as a runtime field on the SampleContainer we might want a field for checking if there are any broken plugins. We will avoid allowing playback or transfer until the plugin is fixed or removed.
For demo sample transform plugins we can have four:
- A simple gain control that allows variable gain adjustment (done by iterating over audio buffer), exposes a gain parameter.
- One demonstrating the use of an embedded third party library Kali timestretch on the Audio Buffer content to implement Varispeed, exposing a speed parameter: https://github.com/Infinity/Kali
- One using Web Audio API to implement a simple reverb with a convolver node
- One using Web Audio API to implement a pipeline of effects (filter, compressor, delay)
So for now I'm testing without enabling the audionode return value, instead it's the user's responsibility to use the web audio api. However I have a handful of problems I need to solve:
- I'm sometimes submitting multiple jobs to the iframe at the same time. I want to keep the iframe persistent across different jobs, so I guess I need to solve this in the app code. One way could be with a short set timeout or similar.
- In some cases it seems like using a plugin can crash the page. I think I can solve this by running it on a different subdomain.
- Sometimes I'm getting an error message that a plugin is running for too long. I usually get this when updating my code in dev mode but I should make sure it's not a regular issue.
- I should probably implement a job queue for each plugin so it doesn't try to handle a bunch of jobs concurrently and make them all exceed their timeouts.
I also need to implement the ability to override sample param values
Other things left to consider are graceful updates when new samples are imported (for names, replacement, params in UI, params in code) to make sure nothing breaks. And we also want to handle batches of many samples sent to the iframe at once (probably)
When we import a new plugin, we save the filename to a variable PLUGIN_NAME
Now we check if the PLUGIN_NAME exists in the store.
label CONFLICT_RESOLUTION: If the a plugin with the name PLUGIN_NAME exists already:
If the content hash matches the existing plugin:
We tell the user that the plugin is already added.
Exit.
If the content hash does NOT match:
We ask if they want to RELACE the existing plugin or else ADD NEW.
If they want to REPLACE:
We have to go through all existing references to PLUGIN_NAME and update their content ids.
Exit.
If they want to ADD NEW:
Update PLUG_NAME to NOT match the previous filename.
Continue.
Now we have a dialog for confirming PLUGIN_NAME and possibly updating it.
Now we check again if a plugin with name PLUGIN_NAME exists.
Go back to CONFLICT_RESOLUTION and repeat as many times as necessary.
For now I'm not implementing the import plugins yet and not supporting non-number params yet. We'll see what feedback there is from users.
What's left to do is:
- I need to remove the duration cached field, which can be computed at runtime, and replace it with a pluginDurationCoefficient which can be multiplied against the duration to get the untrimmed duration of the processed sample. Whenever this coefficient changes we also need to update the trim frames to a rounded new value as a function of the ratio between the old and new coefficients.
- To the sample metadata, besides the coefficient in the cachedData we will want to add a list of plugins. It can include plugin names and plugin params for each plugin.
- The plugins should be included in a subdirectory of the export zip and re-instantiated on import. During import we may need to resolve duplicate plugin names, and if so, we will need to update the plugin names in the samples that use that plugin (unless we choose to replace the plugin).
- When implementing the UI we need to make sure to display a plugin as broken if the getParams promise fails or if onError triggers.
- We need to implement a UI modal for managing the list of added plugins (adding, removing)
- We need to implement UI in the configure section for including or removing plugins on a sample, including sliders with direct input for each parameter (we might want to add the direct input to the other sliders while we're at it?), a dropdown for choosing from the list of available plugins, an option to go to the plugin management view, and some info on what plugins are and how they work. We might want all of this to live in a modal since it would make the main view kind of crowded, then we can show a brief plugin summary with parameters (non configurable) in the main view.
- Plugin configuration (as well as clearing space on the volca) are things that should probably be accessible from other places than just contextual, but it's hard to know where. We need to think about this.
Fatal flaw of "pluginDurationCoefficient" plan is that it's dependent on the result of running the plugins which happens after the metadata changes, so it will cause the audio computation to run in a loop (possibly stopping after two times if we deal with updates intelligently). Also we will have the same problem with waveformPeaks because it will now rely on the plugin result instead of the source data. The simplest solution is to get rid of the cachedData property and store this data separately.
We could have all of this data returned by the getTargetWavForSample function. The src duration is trivial to have since we already read the source buffer (although at this point we still haven't used it, so we could drop it for now). The waveformPeaks can be computed from the plugin result during the function, and the final duration is easy to know by dividing the final samples length by the sample rate.
It's worth thinking about when each of these two pieces of data is needed. The cached duration is needed for estimating transfer time and file size before a transfer (before all the audio is computed). The cached waveformPeaks are needed by the WaveformDisplay (it also needs the normalize boolean, which we could cache along with the cached info so it's consistent).
There are a few times this cached data needs to be saved:
- When we add a new sample (easy)
- When the audio preview gets reloaded for WaveformEdit
- When we bulk import samples
We need to make sure the SampleList knows when cached data is updated, and the thing that computes sample time, but ideally other parts of the UI aren't disturbed. So we might want a React Context for this.
The final sample metadata upgrade should just correct the old duration value by taking the pitch adjustment into account, save it, and pull that info out of the metadata.
So to address the previously mentioned issue, we'll need to create a new store sort of like SampleMetadata but called SampleCache. It will contain the cachedInfo property as well as a method called getTargetWavData (or similar) that either returns a cached copy of the wav data or recomputes it, saves it to the cache and returns it. The audio cache will be similar to the one for source file data. We will construct the object using cachedInfo, and its update method, which is async, will take a SampleMetadata, recompute the wav data, cache it, and return a new object with the new cached info.
The map from sample ID to cachedInfo will be managed at the top of the app like the regular samples. We will no longer use the react hook for getting target WAV data, and the getTargetWavData function should be called only from syro utils and from SampleCache.
A few other notes:
- persist SampleCache only on update, not load
- on sample duplicate we can persist the cached info but we copy the old SampleCache to the new somehow
- we need to deal with deletion as well
- tab sync needs to be accounted for
- we can take this occasion to re simplify the trim property on the metadata
Looks like we can remove the normalizationCoefficient from waveform peaks, it's not always correct anyway. We'll just compute the sample peaks after normalization in the audio computation function.
We need to use a pluginless waveform peaks and duration as a backup when there is an error computing the audio. It's important to handle this case when first creating a sample (which will be the default since we won't have any plugins or pitch adjustment), as well as when importing samples (in this case we will try with plugins first, then fall back if needed).
In the case of an update we want to update the failedPluginIndex on fail but otherwise keep the old cachedInfo.
Before I had mentioned maybe needing a property to flag that plugins failed to run. Instead of including this on sample metadata I will include it as part of cacheddata. It can be set on sample update or on import.
When multi transferring we need to handle an error case for failedPluginIndex.