Enhancement : save the hyperparameters used during training along the checkpoints
Ca-ressemble-a-du-fake opened this issue · comments
Hi,
When trying several hyper parameters and compare the resulting models, it would be useful to have a text file summarizing the hyper parameters used during training. Though there are some edge cases such as when resuming the training from a checkpoint. In that case an updated version of the hyper parameters or better a new file appended to the other files with those new hyper parameters.
Ex training_hyper_params_0.txt then training_hyper_params_1.txt so that the hyper parameter changes can be tracked.
Tell me if you need me to code it
I saw other codebases that save all the hyperparameters as a dictionary inside a checkpoint and then during loading of the checkpoint automatically configure the model architecture based on the information in that dictionary. I think that's a very clean solution to the problem of there being single speaker, multi speaker and multi language versions of the Toucan models and also makes hyperparameter exploration a lot easier. So I think it's a good idea for the future, but right now it's not my priority and I'm not sure yet about the best way to integrate this. It will be in a future version.
You want to be sure it is not altered in anyway, so you want it not to be accessible (edited) by the user. I was thinking more of a text file with an history of the training sessions pertaining to this model (so in the subfolder PortaSpeech_XYZ) with a summary of all hyperparameters (and the date of the training). Like they do in Coqui TTS with their config.json file.
You seem to already have an idea of what you want so I am looking forward to testing this 😊.