akaihola / selim

A real-time musical score follower toolkit

Selim – a real-time musical score follower toolkit

You can provide Selim with

• a MIDI file (or text input with millisecond timestamps and integers for MIDI note-on numbers), and
• a human performance of the same music (on a real-time MIDI device, or as text input via a Unix socket, TCP socket, websocket or standard input)

The software will then do its best to figure out where the player is going in the given MIDI file.

It will be able to output in real time, based on user choices, e.g.:

• a time index in the MIDI file
• events from the original MIDI file, synchronized to the performance
• events from a second MIDI file, synchronized to the performance
• MIDI events from the human performer as pass-through
• ...whatever exciting we'll come up with

The software should be usable as

• a Rust library
• a Python extension
• a command line utility communicating via MIDI devices, Unix sockets, websockets, TCP sockets or standard I/O

Examples

$ cargo run --bin selim-mid-info piece.mid
   Compiling selim v0.1.0 (/home/kaiant/prg/selim)
    Finished dev [unoptimized + debuginfo] target(s) in 0.33s
     Running `target/debug/selim-mid-info 'piece.mid'
midi file has 1 tracks!
first track has 16430 events!
first track has 446 'note on' events on channel 1!

Status and roadmap

• choose MIDI parser library
• ensure we can get data from MIDI files using the chosen MIDI parser (midly)
• data type for an in-memory reference score (milliseconds and pitches)
• first naïve stateless score follower algorithm selim-0.1.0
  • inputs:
    • complete reference score (ms+pitch)
    • complete live input so far (ms+pitch)
    • position of last matching previous input note in the score
    • position of last matching previous input note in the live input
    • position of first new note in the live input
    • time stretch factor at last matching note
  • outputs:
    • reference time index at last new input note (ms)
    • time stretch factor at last new matching note
    • list of ignored new input notes (ms+pitch)
  • support only monophony (order of events matters)
  • ignore unexpected (wrong/extra) notes
  • keep waiting for next correct note
• unit tests for selim-0.1.0
• function to turn a MIDI file into an in-memory reference score (ms+pitch)
  • use only the first track which contains meaningful MIDI data
  • use only the first channel which contains meaningful MIDI data
  • use only "note on" events
  • ignore velocity
  • convert time offsets to milliseconds (disregarding tempo for now)
  • tool to output reference score on stdout
• choose MIDI input library
• real-time tool to convert MIDI input into ms+pitch events on stdout
• real-time tool to test out selim-0.1.x
  • inputs:
    • reference MIDI file
    • real-time ms+pitch events on stdin
  • outputs on stdout:
    • reference time index at last new input note (ms)
    • reference time stretch factor at last new input note
    • ignored input notes
• wrong/missed/extra note tolerant score follower algorithm selim-0.1.1
  • match new input notes with future reference notes within a time window
  • jump directly to first matching note
• time stretch factor adjustment limit in selim-0.1.2
• refine MIDI file interpretation (ms+pitch+vel+dur)
  • take tempo changes into account when converting to milliseconds
  • include velocity
  • convert "note off" events to durations

Resources

• Playing back a midi with midly: how to organise the ecosystem (Rust Audio Discourse group)