ValueError: chr() arg not in range(0x110000)

[JuanCarlosPi]

Hi,

I am trying to tokenize the LMD clean dataset, but it's giving me an error message.

This is my code:

from miditok.tokenizations import remi_plus
# Our parameters
pitch_range = range(21, 109)
beat_res = {(0, 4): 16, (4, 12): 4}
nb_velocities = 32

special_tokens = ["PAD", "BOS", "EOS", "MASK"]

# Creates the tokenizer convert MIDIs to tokens
tokens_path = Path('/content/drive/MyDrive/8 Beats/AI/Datasets/Symbolic/lmd/miditok/no_bpe')
tokenizer = REMIPlus(pitch_range, beat_res, nb_velocities, special_tokens=special_tokens) # MMMM
midi_paths = list(Path('/content/drive/MyDrive/8 Beats/AI/Datasets/Symbolic/lmd/clean_midi').glob('**/*.mid')) + list(Path('/content/drive/MyDrive/8 Beats/AI/Datasets/Symbolic/lmd/clean_midi').glob('**/*.midi'))
tokenizer.tokenize_midi_dataset(midi_paths, tokens_path, validation_fn=midi_valid)

# Learn and apply BPE to data we just tokenized
tokens_bpe_path = Path('/content/drive/MyDrive/8 Beats/AI/Datasets/Symbolic/lmd/miditok/boe')
tokens_bpe_path.mkdir(exist_ok=True, parents=True)
tokenizer.learn_bpe(
    vocab_size=1000,
    tokens_paths=list(tokens_path.glob("**/*.json")),
    start_from_empty_voc=False,
)
tokenizer.apply_bpe_to_dataset(
    tokens_path,
    tokens_bpe_path,
)

And this is the error I get:

[/usr/local/lib/python3.10/dist-packages/miditok/midi_tokenizer.py](https://localhost:8080/#) in tokenize_midi_dataset(self, midi_paths, out_dir, validation_fn, data_augment_offsets, apply_bpe, save_programs, logging)
   1580 
   1581             # Converting the MIDI to tokens and saving them as json
-> 1582             tokens = self(
   1583                 midi, apply_bpe_if_possible=False
   1584             )  # BPE will be applied after if ordered

[/usr/local/lib/python3.10/dist-packages/miditok/midi_tokenizer.py](https://localhost:8080/#) in __call__(self, obj, *args, **kwargs)
   1831         # Tokenize MIDI
   1832         if isinstance(obj, MidiFile):
-> 1833             return self.midi_to_tokens(obj, *args, **kwargs)
   1834 
   1835         # Loads a file (.mid or .json)

[/usr/local/lib/python3.10/dist-packages/miditok/midi_tokenizer.py](https://localhost:8080/#) in midi_to_tokens(self, midi, apply_bpe_if_possible, *args, **kwargs)
    475         }
    476 
--> 477         tokens = self._midi_to_tokens(midi, *args, **kwargs)
    478 
    479         if apply_bpe_if_possible and self.has_bpe:

[/usr/local/lib/python3.10/dist-packages/miditok/tokenizations/remi_plus.py](https://localhost:8080/#) in _midi_to_tokens(self, midi, *args, **kwargs)
    350         """
    351         # Convert each track to tokens
--> 352         events = self.__notes_to_events(midi.instruments)
    353         tok_sequence = TokSequence(events=cast(List[Union[Event, List[Event]]], events))
    354         self.complete_sequence(tok_sequence)

[/usr/local/lib/python3.10/dist-packages/miditok/tokenizations/remi_plus.py](https://localhost:8080/#) in __notes_to_events(self, tracks)
    129             if self.max_bar_embedding < nb_bars:
    130                 for i in range(self.max_bar_embedding, nb_bars):
--> 131                     self.add_to_vocab(f"Bar_{i}")
    132                 self.max_bar_embedding = nb_bars
    133         current_bar = -1

[/usr/local/lib/python3.10/dist-packages/miditok/midi_tokenizer.py](https://localhost:8080/#) in add_to_vocab(self, token, vocab_idx, byte_, add_to_bpe_model)
    768             # For BPE
    769             if byte_ is None:
--> 770                 byte_ = chr(id_ + CHR_ID_START)
    771             self._vocab_base_id_to_byte[
    772                 id_

ValueError: chr() arg not in range(0x110000)

Thanks again for the great library!

[Nathan Fradet]

Hi,

This error happens because the tokenizer is trying to add a new character / token to the vocabulary. This happens for Bar tokens, when the tokenizer is processing à MIDI with an overall duration that is longer that the nb of bars in the vocabulary.
But here 0x110000 (roughly 1M in decimal) is abnormally high (like a 1M bar long MIDI…). I think the MIDI file might be corrupted. I suggest you to preprocess the MIDI files, and cut each of them by chunks of $N$ bars.
Or alternatively, set max_bar_embedding=False when creating the tokenizer, to get rid of this issue

[JuanCarlosPi]

Hi @Natooz! As always, thanks for your quick response. Sounds great, I'll try what you mention. Would you know if there's an ready-to-use tool for creating the chunks? Otherwise, I'll try to create the code for this.

Thanks again.

[Nathan Fradet]

Yes, you can take this snippet that I wrote some time ago

from pathlib import Path
from copy import deepcopy
from math import ceil

from miditoolkit import MidiFile
from tqdm import tqdm


MAX_NB_BAR = 20
MIN_NB_NOTES = 20
dataset = "MMD"

merged_out_dir = Path("data", f"{dataset}-chunked")
merged_out_dir.mkdir(parents=True, exist_ok=True)
midi_paths = list(Path("data", dataset).glob("**/*.mid"))

for i, midi_path in enumerate(tqdm(midi_paths, desc="CHUNKING MIDIS")):
    # Loads MIDI, merges and saves it
    midi = MidiFile(midi_path)
    ticks_per_cut = MAX_NB_BAR * midi.ticks_per_beat * 4
    nb_cuts = ceil(midi.max_tick / ticks_per_cut)
    if nb_cuts < 2:
        continue
    midis = [deepcopy(midi) for _ in range(nb_cuts)]

    for j, track in enumerate(midi.instruments):  # sort notes as they are not always sorted right
        track.notes.sort(key=lambda x: x.start)
        for midi_short in midis:  # clears notes from shorten MIDIs
            midi_short.instruments[j].notes = []
        for note in track.notes:
            cut_id = note.start // ticks_per_cut
            note_copy = deepcopy(note)
            note_copy.start -= cut_id * ticks_per_cut
            note_copy.end -= cut_id * ticks_per_cut
            midis[cut_id].instruments[j].notes.append(note_copy)

    # Saving MIDIs
    for j, midi_short in enumerate(midis):
        if sum(len(track.notes) for track in midi_short.instruments) < MIN_NB_NOTES:
            continue
        midi_short.dump(merged_out_dir / f"{midi_path.stem}_{j}.mid")

[github-actions]

This issue is stale because it has been open for 30 days with no activity.