SP-Tools - Machine Learning tools for low latency real-time performance

SP-Tools are a set of machine learning tools that are optimized for low latency and real-time performance. The tools can be used with any audio input but include optimizations for Sensory Percussion sensors and ordinary drum triggers.

SP-Tools includes low latency onset detection, onset-based and real-time descriptor analysis, classification and clustering, corpus analysis and querying, concatenative synthesis, neural network predictive regression, and a slew of other abstractions that are optimized for low latency applications.

SP-Tools is built around the FluCoMa Toolkit and requires v1.0.6+ to be installed for this package to work.

Discord Server

You can join the SP-Tools Discord server to keep up on future updates/news as well as make suggestions, share corpora and musical examples etc...
https://discord.gg/Vy2zuKQkqN

Requirements

Max 8.3 or higher or Live/M4L (Mac/Windows).
FluCoMa v1.0.6 or higher - Download Link
All abstractions work in 64-bit and M1/Universal Binary 2.

Useful Videos

SP-Tools Teaser Video - Performance and Musical Examples
SP-Tools (alpha v0.1) - Initial Overview
SP-Tools (alpha v0.2) - Controllers and Setups
SP-Tools (alpha v0.3) - Filtering, Playback, and Realtime Analysis
SP-Tools (alpha v0.4) - Concatenation and Realtime Filtering
SP-Tools (alpha v0.5) - Grid-Based Matching, Erae Touch, and Max for Live
SP-Tools (alpha v0.6) - Max for Live Walkthrough
SP-Tools (alpha v0.7) - Ramps, Data Processing, Novelty, and Timestretching
SP-Tools (alpha v0.8) - Sines, Synthesis/Modelling, and Documentation
SP-Tools (alpha v0.9) - Scope, Ease of Use, and Audio Processing

Changelog

v0.9 - Scope, Ease of Use, and Audio Processing

BREAKING CHANGE - objects with "train" in the name (sp.classtrain, sp.clustertrain, sp.setuptrain~) have been renamed to be more consistent with other objects (sp.classcreate, sp.clustercreate, sp.setupcreate~)
BREAKING CHANGE - sensitivity parameter in all onset-based objects has been fine-tuned to work across a wider range of instruments and materials so your previous settings may likely need adjusting
BREAKING CHANGE - threshold parameter renamed to floor in all onset-based objects (sp.onset~, sp.onsetframe, sp.descriptors~, sp.melbands~, sp.mfcc~, sp.sines~, etc...)
BREAKING CHANGE - threshold parameter renamed to sensitivity in sp.novelty~ keeping it inline with other onset detection objects macro controls (sp.onset~, sp.spectralonset~)
BREAKING CHANGE - mode parameter renamed to timestretchmode in sample playback objects keeping to remove clash with the new mode attribute
BREAKING CHANGE - input mode order and modes changed on all onset-based object (sp.onsetframe, sp.descriptors~, sp.melbands~, sp.mfcc~, sp.sines~, etc...)
BREAKING CHANGE - clusters trained with sp.clustertrain now count from 1 instead of 0 and are returned as ints instead of a symbols (so a 1 instead of a "1")
BREAKING CHANGE - sp.classifierdisplay @drum attribute reworked to @defaultdisplay
BREAKING CHANGE - fade in/out and curve parameters renamed in sp.corpusplayer~ to be consistent with the new sp.sampler~ (attack=in, hold=out(and inverted), attackcurve=incurve, holdcurve=outcurve)
overview patch (SP-Tools Overview.maxpat) completely revamped with better discoverability and package navigation
added snippets for common use cases to SP-Tools package (accessible via the snippets side panel)
restructured Max Package folders to make it easier to navigate and add additional corpora
restructured github repo to separate versions (Max, pd), Max for Live devices, and additional corpora
added class-combined versions of all descriptor analysis objects making it easier to get classification combined with descriptor analysis (sp.classdescriptors~, sp.classmfccs~, etc...)
added sp.controllermatch to browse corpora using a single parameter
added sp.controllerptich to create pitch-quantized melodic phrases from a single parameter
added zero-latency convolution (sp.convolver~) which allows for reverb, amp sim, mic correction, and any other convolution-based audio processes
added convolution-based corpus matching via sp.corpusconvolver~ which combines corpus matching and realtime convolution
added combined abstractions for common processes (sp.corpusanalysis~, sp.corpussampler~)
substantially improved core concatenation algorithm
added radius and neighbor selection in concat matching
added pitch and loudness compensation to concat playback
added transpose and pretranspose to concat playback
added sp.corpusclustermatch for creating clusters from corpora, allowing for grouping of similar sounds within a corpus (e.g. kick, snare, hat, etc...)
added objects for shaping and processing descriptor lists (sp.descriptorcurves, sp.descriptorreplace)
added an envelope follower with some quirky features (sp.envelope~)
added sp.gridscale for scaling and shifting XY-based controller inputs (combines well with sp.gridmatch and the new sp.boppad)
added pitch and time quantization with sp.quantizepitch and sp.quantizetime~
added data and clock sequencers with sp.sequenceclock~ and sp.sequencedata~
added Schmitt filter for descriptor filtering for when you need hysterisis in the filtering (sp.schmitt)
added various audio processes for splitting and decomposing signals (sp.scatter~, sp.scramble~, sp.shatter~, sp.shattercreate~, sp.sift~, sp.smear)
added spectral-difference onset detection (sp.spectralonset~)
added new kitchensink descriptor type, as a combination of descriptor/melband/MFCC buffers which comes in onset, real-time, and class-based variants (sp.kitchensink~, sp.kitchensinkrt~, sp.classkitchensink~)
added neural network classifier type to sp.classtrain, sp.clustertrain, and sp.classmatch for greatly improved speed and accuracy
added support for the BopPad controller (sp.boppad)
added input modes to all the physical modeling objects (sp.karplus~, sp.resonators~, sp.shaker~, sp.waveguidemesh~) allowing for realtime actuation of the models, in addition to onset/trigger-based actuation
added ability to reverse and loop samples, and jump directly to sample position in sp.corpusplayer~
added @mode attribute to sp.onsetframe~ to switch between amplitude and spectral onset detection (also impacts all onset-based descriptors objects (sp.descriptors~, sp.melbands~, sp.mfcc~, sp.sines~, etc...))
added ability to seed clusters in sp.clustertrain
added utility for subdividing a spaces of arbitrary size and steps (sp.slicecurve)
added utility for time-alining trigger/gates with descriptor output (sp.triggeralign~)
added utility for increasing the length of triggers and gates (sp.triggerlength~)
added AHR-style envelope with sp.triggershape~
improved time-alignment for trigger/gate outputs of descriptors objects (sp.descriptors~, sp.melbands~, sp.mfcc~, sp.sines~, etc...)
reworked sp.classifierdisplay to automatically load class names from all class matching objects
reworked sp.descriptordisplay to automatically display the correct type of descriptor
added dump output message to all class-based objects (sp.classmatch, sp.classdescriptors, etc...)
added mic correction (via a new input mode attribute @input 5) to all onset-based descriptor analysis objects for better descriptor results when using only an SP sensor
added @buffer attribute to sp.corpusplayer~ to simplify non corpus-based sample playback
added @history attributes to all realtime descriptor objects (sp.descriptorsrt~, sp.melbandsrt~, sp.mfccrt~, sp.sinesrt~) to smooth descriptors
added @input modes to sp.onset~ allowing for optional filters optimized for different input types
added onthreshold and offthreshold to sp.novelty~ given you finer grain control of thresholding
improved (and fixed) melband transposition in sp.datatranspose
added optional arguments to all corpus matching abstractions to load corpus and setups automatically (e.g. sp.corpusmatch corpus_china setup_snare will automatically load the China Cymbal corpus and load/enable the snare setup)
added ability to load default files to all .json loading objects (e.g. sp.resonators~ resonators_bell)
added header to every .json file to differentiate them (you should reanalyze all your corpora, classes, and setups)
when loading a setup into any corpus matching abstraction (sp.corpusmatch, sp.corpusconvolver~, sp.corpussampler~) it will automatically enable that setup
improved default starting values in sp.controllers so it gives useful results while it autocalibrates
added ability to trigger sp.triggerbounce~ and sp.triggercloud~ with bangs as well as trigger/gates
added ability to trigger synthesis objects (sp.lpg~, sp.karplus~, sp.resonators~, sp.waveguidemesh~) with bangs as well as triggers/gates
added 8 more abstractions to the puredata version
changed FFT settings for medium and all sizes for corpus and setup analysis to have better pitch accuracy in the larger time scales. You should re-analyze all your own corpora and setups.
you can now load mp3 files into sp.corpuscreate
fixed M4L device loading bug(s)
abstractions that only function as part of an existing abstraction (sp.lpgcore~, sp.folderloop, etc...) have had their sp. prefix replaced with widget and have had their help/reference pages removed

v0.8 - Sines, Synthesis/Modelling, and Documentation

added synthesis and physical modelling objects (sp.karplus~, sp.resonators~, sp.resonaroscreate~, sp.shaker~, sp.sinusoidplayer~, sp.sinusoids~, sp.waveguidemesh~, sp.lpg~, sp.lpgcore~)
added Max for Live devices for some of the new processes (Resonators, Sinusoids, Waveguide Mesh)
added new descriptor type (sines) and corresponding objects (sp.sineframe, sp.sines~, sp.sinesrt~) (you should reanalyze all your corpora)
added utilities for filtering and creating clusters of triggers and gates (sp.probability~, sp.triggerbounce~, sp.triggercloud~, sp.triggerframe~)
added absolute start and length parameters to sp.corpusplayer~ (via startabsolute and lengthabsolute messages)
added scramble transformation to sp.databending
added slew parameter to sp.speed
added ability to loop floats and ints (as well as conventional descriptors) to sp.datalooper~
added proper Max documentation via reference files, autocomplete, etc...
added sample rate adaptation to all realtime and offline analyses. previously things were optimized and assumed for 44.1k/48k but now everything works at every sample rate (up to 192k)
added some puredata abstractions and help files to the package (in the puredata folder)

v0.7 - SP-Tools v0.7 - Ramps, Data Processing, Novelty, and Timestretching

BREAKING CHANGES - all objects that had a separate control inlet, now take those messages in the left-most inlet
added new "ramp" objects for structural and gestural changes (sp.ramp, sp.ramp~)
added new "data" objects for transforming, looping, and delaying descriptors (sp.databending, sp.datadelay, sp.datagranular, sp.datalooper~, sp.datatranspose)
added novelty-based segmentation for determining changes in material type (sp.novelty~)
added timestretching functionality to sp.corpusplayer~ and the Corpus Match M4L device

v0.6 - SP-Tools v0.6 - Max for Live Walkthrough

added Max for Live devices (16 total) which cover (nearly) all the functionality of SP-Tools
Max codebase further commented and tidied

v0.5 - SP-Tools v0.5 - Grid-Based Matching, Erae Touch, and Max for Live

added Max for Live devices for some of the main/flagship functionality (Concat Match, Controllers, Corpus Match, Descriptors, Speed)
added sp.gridmatch abstraction for generic controller-based navigation of corpora
added support for the Erae Touch controller (sp.eraetouch)
improved path stability when loading example corpora

v0.4 - SP-Tools v0.4 - Concatenation and Realtime Filtering

added "concat" objects for real-time mosaicking and concatenative synthesis (sp.concatanalysis~, sp.concatcreate, sp.concatmatch, sp.concatplayer~, sp.concatsynth~)
added ability to apply filtering to any descriptor list (via sp.filter)
improved filtering to allow for multiple chained criteria (using and and or joiners)
updated/improved pitch and loudness analysis algorithms slightly (you should reanalyze corpora/setups/etc...)

v0.3 - SP-Tools v0.3 - Filtering, Playback, and Realtime Analysis

added ability to filter corpora by descriptors (baked into sp.corpusmatch via filter messages)
added improved/unified corpus playback with sp.corpusplayer~
add realtime analysis abstractions (sp.realtimeframe~, sp.descriptorsrt~, sp.melbandsrt~, sp.mfccrt~)
added new stereo corpus (corpus_plumbutter.json)
improved corpus analysis to work with stereo files and files shorter than 100ms as well as adding more comprehensive metadata
added sp.corpuslist abstraction for visualizing and playing samples in a corpus in list form
removed old playback abstractions (sp.corpussimpleplayer~, sp.corpusloudnessplayer~, sp.corpusmelbandplayer~)

v.02 - SP-Tools v0.2 - Controllers and Setups

added "setups" (corpus scaling and neural network prediction/regression)
added "controllers" (meta-parameters extracted from onset timings and descriptor analysis)
added four new abstractions (sp.controllers, sp.speed, sp.setupanalysis, sp.setuptrain~)
added new corpus (corpus_voice.json)
added @roundrobin mode to sp.corpusmatch

rconstanzo / SP-tools