How much compute to evaluate?

[Peter Greis]

I have just attempted to run the eval code on an A6000. While it starts, it looks like it just hangs. Hence my question - what did you run the eval code on? Same cluster as mentioned in the paper for training?

[hf-lin]

We run eval code with an RTX4090 GPU. Actually, 14GB memory is enough for a 7B model to inference.

[Peter Greis]

And to add our fine tuned models to the set is it simply modify: eval/configs/models/chat_musician/hf_chat_musician.py ?

Something like

model_path_mapping = {
    "ChatMusician": "m-a-p/ChatMusician",
    "ChatMusician-Base": "m-a-p/ChatMusician-Base",
    "Mozart-SImple": "/full/path/to/model",
    "Mozart-Transposed": "/full/path/to/model_transposed"
}

??