about diffVC on Mandarin datasets
Theweekfoolish229 opened this issue · comments
Hello, I adapted the diffvc code on Mandarin datasets. However, the audio after VC has the problem of tone sandhi. I want to ask the performance is normal ?
It's diffvc performance on Mandarin datasetshttps://www.yuque.com/qinjitao/te3orn/zfgbp9
Hello, I adapted the diffvc code on Mandarin datasets. However, the audio after VC has the problem of tone sandhi. I want to ask the performance is normal ?
Hi, I meet the same problem, did you solve it？
Hi, @Theweekfoolish229 !
Sorry to hear about that. Actually, we tested our voice conversion model only on the English dataset, and even for this language, despite the overall good quality, there were some problems with the source prosody preservation (including mispronunciation issues as discussed in Section 4.2 of our paper). This is because our DiffVC model has only the so-called "average mel-spectrograms" as the information about the content of the source utterance. For tone languages like Mandarin it may be insufficient.
However, in our recent paper we proposed a sampling method based on the optimal transport property that keeps source prosody better compared to vanilla sampling from a diffusion model. Although we didn't test it on Mandarin, we suppose it should work better on it as well and help to reduce the problem with tones. I hope that in the nearest future we'll add the sampling method described in the mentioned paper to our repo.