ttslearnにおける転移学習について
mule-engineer13 opened this issue · comments
初めまして。mule-engineer13と申します。
『Pythonで学ぶ音声合成』を参考に音声合成の学習を進めております。
1点質問をさせていただきます。
ttslearnのtacotron2とwavenetボコーダについて、転移学習を行うことは可能でしょうか？
可能であれば、方法を伺えますと幸いです。
何卒よろしくお願いいたします。
回答が遅くなりすみません。可能です。リポジトリにソースコードが含まれていますので、下記のコードを参考にしていただければと思います。
レシピ：
ttslearn/extra_recipes/jvs/multispk_tacotron2_pwg/run.sh
Lines 146 to 190 in ea75a16
| if [ ${stage} -le 3 ] && [ ${stop_stage} -ge 3 ]; then | |
| echo "stage 3: Training Tacotron" | |
| if [ ${finetuning} = "true" ] && [ -z ${pretrained_acoustic_checkpoint} ]; then | |
| pretrained_acoustic_checkpoint=$PWD/../../jsut/tacotron2_pwg/exp/jsut_sr${sample_rate}/${acoustic_model}/${acoustic_eval_checkpoint} | |
| if [ ! -e $pretrained_acoustic_checkpoint ]; then | |
| echo "Please first train a acoustic model for JSUT corpus!" | |
| echo "Expected model path: $pretrained_acoustic_checkpoint" | |
| exit 1 | |
| fi | |
| fi | |
| xrun python train_tacotron.py model=$acoustic_model tqdm=$tqdm \ | |
| data.train.utt_list=data/train.list \ | |
| data.train.in_dir=$dump_norm_dir/$train_set/in_tacotron/ \ | |
| data.train.out_dir=$dump_norm_dir/$train_set/out_tacotron/ \ | |
| data.dev.utt_list=data/dev.list \ | |
| data.dev.in_dir=$dump_norm_dir/$dev_set/in_tacotron/ \ | |
| data.dev.out_dir=$dump_norm_dir/$dev_set/out_tacotron/ \ | |
| train.out_dir=$expdir/${acoustic_model} \ | |
| train.log_dir=tensorboard/${expname}_${acoustic_model} \ | |
| train.max_train_steps=$tacotron_train_max_train_steps \ | |
| data.batch_size=$tacotron_data_batch_size \ | |
| cudnn.benchmark=$cudnn_benchmark cudnn.deterministic=$cudnn_deterministic \ | |
| train.pretrained.checkpoint=$pretrained_acoustic_checkpoint | |
| fi | |
| if [ ${stage} -le 4 ] && [ ${stop_stage} -ge 4 ]; then | |
| echo "stage 4: Training Parallel WaveGAN" | |
| if [ ${finetuning} = "true" ] && [ -z ${pretrained_vocoder_checkpoint} ]; then | |
| voc_expdir=$PWD/../../jsut/tacotron2_pwg/exp/jsut_sr${sample_rate}/${vocoder_model} | |
| pretrained_vocoder_checkpoint="$(ls -dt "$voc_expdir"/*.pkl | head -1 || true)" | |
| if [ ! -e $pretrained_vocoder_checkpoint ]; then | |
| echo "Please first train a PWG model for JSUT corpus!" | |
| echo "Expected model path: $pretrained_vocoder_checkpoint" | |
| exit 1 | |
| fi | |
| extra_args="--resume $pretrained_vocoder_checkpoint" | |
| else | |
| extra_args="" | |
| fi | |
| xrun parallel-wavegan-train --config $parallel_wavegan_config \ | |
| --train-dumpdir $dump_norm_dir/$train_set/out_tacotron \ | |
| --dev-dumpdir $dump_norm_dir/$dev_set/out_tacotron/ \ | |
| --outdir $expdir/$vocoder_model $extra_args | |
| fi |
Pythonコード:
ttslearn/ttslearn/contrib/multispk_util.py
Lines 156 to 177 in ea75a16
| # (optional) 学習済みモデルの読み込み | |
| # ファインチューニングしたい場合 | |
| pretrained_checkpoint = config.train.pretrained.checkpoint | |
| if pretrained_checkpoint is not None and len(pretrained_checkpoint) > 0: | |
| logger.info( | |
| "Fine-tuning! Loading a checkpoint: {}".format(pretrained_checkpoint) | |
| ) | |
| checkpoint = torch.load(pretrained_checkpoint, map_location=device) | |
| state_dict = checkpoint["state_dict"] | |
| model_dict = model.state_dict() | |
| state_dict = {k: v for k, v in state_dict.items() if k in model_dict} | |
| invalid_keys = [] | |
| for k, v in state_dict.items(): | |
| if model_dict[k].shape != v.shape: | |
| logger.info(f"Skip loading {k}") | |
| invalid_keys.append(k) | |
| for k in invalid_keys: | |
| state_dict.pop(k) | |
| model_dict.update(state_dict) | |
| model.load_state_dict(model_dict) | |
ご回答ありがとうございます！
早速試してみます。