########################################################################### ## prodject-CURRENNT-scrits ------------------------------------------- # ## --------------------------------------------------------------------- # ## # ## Copyright (c) 2020 National Institute of Informatics # ## # ## THE NATIONAL INSTITUTE OF INFORMATICS AND THE CONTRIBUTORS TO THIS # ## WORK DISCLAIM ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, INCLUDING # ## ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS, IN NO EVENT # ## SHALL THE NATIONAL INSTITUTE OF INFORMATICS NOR THE CONTRIBUTORS # ## BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY # ## DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, # ## WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS # ## ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE # ## OF THIS SOFTWARE. # ########################################################################### ## Author: Xin Wang # ## Date: 2016 - 2020 # ## Contact: wangxin at nii.ac.jp # ########################################################################### This repository contains the scripts to use CURRENNT \- waveform-modeling: scripts for waveform models \- acoustic-modeling: scripts for acoustic models For NSF waveform-models, pytorch re-implementation is available now: https://github.com/nii-yamagishilab/project-NN-Pytorch-scripts ----------------- 0. before using ---------------------- Please install CURRENNT and pyTools before using this repository. https://github.com/nii-yamagishilab/project-CURRENNT-public Please also install sox http://sox.sourceforge.net Please modify the environment variables in ./init.sh Please read the intruction below to use this repository ----------------- 1. waveform models -------------------- ./waveform-modeling | - DATA Directory to store the data for model training (validation will be selected automatically from these data) - TESTDATA Directory to store the data for test - TESTDATA-for-pretrained Directory to store the data for test for the project-WaveNet-pretrained and project-NSF-pretrained - SCRIPTS Scripts of the training/generation processes - project-NSF-pretrained Neural source-filter model trained on SLT. This is a demonstration script to generate waveforms from a trained NSF model. The samples have been uploaded to https://nii-yamagishilab.github.io/samples-nsf/nsf-v1.html Note: due to historical reason, meanstd.bin for these pre-trained NSF models were not calculated using public-CURRENNT-scripts. Models project-NSF-pretrained can only use project-NSF-pretrained/meanstd.bin If you want to train models on your own corpus, please use the training scripts in project-NSF - project-NSF-v2-pretrained Simplified and harmonic-plus-noise (hn-sinc) NSFs trained on SLT. This is a demonstration script to generate waveforms from a trained NSF model. These new NSF models are explained here: https://nii-yamagishilab.github.io/samples-nsf/nsf-v2.html, https://nii-yamagishilab.github.io/samples-nsf/nsf-v3.html If you want to train models on your own corpus, please use the training scripts in project-NSF - project-NSF-pretrained-v4-CMU-4speakers hn-sinc-NSF using different types of source signals, including cyclic-noise-based source. All models trained on CMU-arctic using CLB, RMS, BDL, and SLT in a speaker-independent wavy. This is a demonstration script to generate waveforms from a trained NSF model. These new NSF models are explained here: https://nii-yamagishilab.github.io/samples-nsf/nsf-v4.html, If you want to train models on your own corpus, please use the training scripts in project-NSF - project-NSF-pretrained-v4-VCTK hn-sinc-NSF and cyclic-noise-based hn-sinc-NSF (see protocol in downloaded TESTDATA-for-pretrained-vc-VCTK after running 00_gen_from_pretrained_models.sh) These new NSF models are explained here: https://nii-yamagishilab.github.io/samples-nsf/nsf-v4.html - project-NSF Scripts to train NSF on CMU-arctic SLT voice. Note that project-NSF/MODELS contains both network.jsn for models in both project-NSF-pretrained/MODELS and project-NSF-v2-pretrained/MODELS: ./project-NSF/MODELS |- s-NSF: simplified NSF, which is used in project-NSF-v2-pretrained/MODELS/s-NSF |- h-NSF: harmonic-plus-noise NSF, which is used in project-NSF-v2-pretrained/MODELS/h-NSF |- h-sinc-NSF: h-NSF with trainable filters, which is used in project-NSF-v2-pretrained/MODELS/h-sinc-NSF |- cyclic-noise-h-sinc-NSF, h-sinc-NSF using cyclic-noise-source, which is used in /project-NSF-pretrained-v4-CMU-4speakers/MODELS/05_beta2/ |- NSF: baseline NSF, which is used in project-NSF-pretrained/MODELS/NSF |- NSF-L3, NSF-MSE, NSF-N2, NSF-S3, which used in project-NSF-pretrained/MODELS/NSF-* You can find other model definition files called network.jsn in pre-trained model directories. - project-WaveNet-pretrained Wavenet trained on SLT. This is a demonstration script to generate waveforms from a trained WaveNet model. The samples have been uploaded to https://nii-yamagishilab.github.io/samples-nsf/nsf-v1.html - project-WaveNet-pretrained-v4-CMU-4speakers Wavenet trained on SLT, BDL, CLB, RMS. This is a demonstration script to generate waveforms from a trained WaveNet model. The samples have been uploaded to https://nii-yamagishilab.github.io/samples-nsf/nsf-v4.html - project-WaveNet-pretrained-v4-VCTK WaveNet trained on VCTK (see protocol in downloaded TESTDATA-for-pretrained-vc-VCTK after running 00_gen_from_pretrained_models.sh) The samples have been uploaded to https://nii-yamagishilab.github.io/samples-nsf/nsf-v4.html - project-WaveNet Scripts to train WaveNet on CMU-arctic SLT voice. Usage: 1. Install CURRENNT and pyTools, which can be downloaded from https://github.com/nii-yamagishilab/project-CURRENNT-public 2. For a quick check 2.1 modify the path in ./init.sh 2.2 run commands: $: source ./init.sh $: cd waveform-modeling/project-NSF-pretrained/ $: run 01_gen.sh Waveforms should be generated in ./waveform-modeling/project-NSF-pretrained/MODELS/NSF/output 2. For model training using the provided sample data $: source ./init.sh $: cd waveform-modeling/project-NSF/ $: run 00_run.sh After which you can get a trained model in ./waveform-modeling/project-NSF/MODELS/h-sinc-NSF/trained_network.jsn $: run 01_gen.sh After which you can get some waveforms in ./waveform-modeling/project-NSF/MODELS/h-sinc-NSF/output 00_run.sh and 01_gen.sh are only for demonstration. Please don't expect good output since the model is only trained using less than 10 utterances. To train a good model, you may need to use the whole data from one speaker of the CMU-arctic corpus. 3. For training using your own data (or CMU-arctic data): 1. Put waveforms and acoustic features in ./DATA, which stores the training data (validation data will be automatically selected from ./DATA) 2. Read and configure config.py in project-NSF or project-Wavenet 3. Run 00_run.sh 4. Put test data in ./TESTDATA, configure config.py and Run 01_gen.sh ----------------- 2. acoustic models -------------------- ./acoustic-modeling |- DATA Directory to store the data for model training (for demonstration) - TESTDATA Directory to store the data for test (for demonstration) - project-DAR-continuous Project scripts to train a DAR for continuous-valued output features. The demonstration is for training a DAR model that does ppg, xvector, f0 -> Mel-spectrogram - project-RNN Project scripts to train a RNN for continuous-valued output features. The demonstration is for training an RNN that does linguistic_features, one-hot_speaker_vector -> MGC, LF0, UV, BAP - SCRIPTS General scripts of the training/generation processes. Usage: please read README in project-***, and follow the steps to use ---------------------- Notes ----------------------------------- -------- Data IO: 1. All the feature files except waveforms should be saved as binary, float32, little-endian. You may check the data included in waveform-modeling/TESTDATA-for-pretrained/mfbsp/*. $: source ./init.sh $: python >>> from ioTools import readwrite >>> mel = readwrite.read_raw_mat('./waveform-modeling/TESTDATA-for-pretrained/mfbsp/arctic_a0001.mfbsp', 80) >>> mel.shape (671, 80) >>> mel[0] array([-0.5804557 , 0.64407444, 0.95468473, 0.9076864 , 0.7409921 , 0.5190042 , 0.2543492 , 0.07272982, -0.02690054, -0.08740515, -0.14761803, -0.24591875, -0.40614623, -0.49193513, -0.69396436, -0.6916913 , -0.67114395, -0.7134891 , -0.8428167 , -0.9381612 , -0.9604182 , -1.020919 , -1.0435845 , -1.077764 , -1.1341914 , -1.2459651 , -1.2637303 , -1.3517176 , -1.3730341 , -1.4908298 , -1.5803422 , -1.6830492 , -1.495914 , -1.3974593 , -1.4675162 , -1.5840678 , -1.6725454 , -1.584573 , -1.7740629 , -2.0816884 , -1.8013108 , -1.7561418 , -2.1837492 , -2.3368633 , -2.0934896 , -2.2621613 , -2.200103 , -2.3064234 , -1.9597349 , -2.2220097 , -2.296505 , -2.0561726 , -2.4601874 , -1.997487 , -2.0367327 , -2.2498186 , -2.8739617 , -2.859662 , -2.680149 , -3.1865113 , -3.172631 , -2.8548572 , -3.0105433 , -2.7395592 , -2.7438028 , -2.625576 , -2.8859007 , -2.8262408 , -2.6442852 , -2.847031 , -2.9952297 , -2.9672284 , -2.6831682 , -3.2138064 , -3.3470006 , -3.4002392 , -2.8704414 , -2.958755 , -3.2552214 , -3.5245233 ], dtype=float32) >>> mel[0,0] -0.5804557 >>> f0 = readwrite.read_raw_mat('./waveform-modeling/TESTDATA-for-pretrained/f0/arctic_a0001.f0', 1) >>> f0.shape (671,) Here, the binary mel-spectrogram is a matrix with 671 frames and 80 dimenions/frame. The F0 is one-dimensional vector with 671 frames. Notice that in physical memory, one datum in a two dimensional data matrix (e.g., mel-spectrom) is accessed through DataArray[D * n + d], where D is the feature dimension, n is the frame index, and d is the dimension index within one frame. 2. You can use numpy.tofile to write the data into binary,float32,litten-endian format. You can also use the write_raw_mat function in pyTools, which is a wrapper of numpy.tofile $: source ./init.sh $: python >>> from ioTools import readwrite >>> import numpy as np >>> data = np.random.randn(10,2) >>> data array([[-1.0792218 , -2.00836936], [-0.84080859, 1.81592092], [ 0.48318553, -0.76937456], [-1.90552536, -0.68052287], [-0.72223174, -2.97435219], [ 0.19932163, 0.29391472], [ 0.36049151, 0.64871376], [ 0.73407896, 0.6574951 ], [ 0.5137015 , -0.67185778], [-0.62208806, -0.35707845]]) # write data to './tmp_data.bin' >>> readwrite.write_raw_mat(data, 'tmp_data.bin') True # read data from './tmp_data.bin' >>> data_read =readwrite.read_raw_mat('tmp_data.bin', 2) # data_read should be the same to data # (except for the nuemrical reason due to the conversion from # float64 to float32) >>> data_read array([[-1.0792218 , -2.0083694 ], [-0.8408086 , 1.815921 ], [ 0.48318553, -0.76937455], [-1.9055253 , -0.68052286], [-0.72223175, -2.9743521 ], [ 0.19932163, 0.2939147 ], [ 0.3604915 , 0.64871377], [ 0.73407894, 0.6574951 ], [ 0.5137015 , -0.6718578 ], [-0.6220881 , -0.35707846]], dtype=float32) -------- Useful CURRENNT commands 1. Continue training using epoch***.autosave $: currennt --continue epoch***.autosave If autosave = true is configured in train_config.cfg, CURRENNT will save trained models after every training epoch. The name of the saved model will be epoch***.autosave. If the training is terminated for any reason, you can simply use the above command to resume the training from the lastes training epoch. No additional argument is needed because epoch***.autosave saves all the arguments, weights, intermediate gradients ... 2. Convert ***.autosave to ***.jsn $: currennt --print_weight_to epoch***.jsn --print_weight_opt 2 --cuda off --network epoch***.autosave ***.autosave is the trained model after *** epochs. It can be used as a trained network to generate output. However, ***.autosave training arguments, gradients, etc, which makes ***.autosave very large. ***.jsn is the trained model, without data unnecessary for generation. To save space, use the above command to convert ***.auto to ***.jsn. Note that network.jsn usually denotes the initial network without any trained weight. 3. Plot the network typology $: currennt --network_graph network.gv --network network.jsn --cuda off $: dot -Tpdf -o network.pdf network.gv Step1 converts network.jsn to network.gv, a file in dot-language Step2 uses "dot" to produce a picture given the network.gv You can check ./acoustic-modeling/project-DAR-continuous/MODELS/DAR_001/network.pdf 4. For debugging, you can use $: gdb --args currennt *** where *** is the argument string to CURRENNT. To compile a version of currennt for debugging, please check README in https://github.com/nii-yamagishilab/project-CURRENNT-public/tree/master/CURRENNT_codes ---------- Error messages 1. Memory error when using CURRENNT 'thrust::system::system_error' what(): device free failed: an illegal memory access was encountered Aborted (core dumped) or Could not create layer: __copy::trivial_device_copy H->D: failed: invalid argument The two errors above indicate insufficient GPU memory. Either reduce layer size, utterance length, or ask Xin Wang 2. Scripts configuration error: resolution not found in --resolutionsFAILED: Resolution error This means that upsampling_rate in wavefor-modeling/*/config.py is not correctly configured. Note that this upsampling_rate denotes the up-sampling of acoustic features (from frame-level to waveform-sampling-level), its value should be equal to frame_shift * sampling_rate_of_waveform. See config.py for example