Giters

keonlee9420 / Parallel-Tacotron2

PyTorch Implementation of Google's Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

LightWeightConv layer warnings during training

idcore opened this issue · comments

commented

If just install specified requirements + Pillow and fairseq following warnings appear during training start:

No module named 'lightconv_cuda'

If install lightconv-layer from fairseq, the folllowing warning displayed:

WARNING: Unsupported filter length passed - skipping forward pass

Pytorch 1.7
Cuda 10.2
Fairseq 1.0.0a0+19793a7

commented

@idcore
I also have a same problem.
Just installing fairseq does not solve this problem.
How were you able to install lightconv_layer independently?

commented

If just install specified requirements + Pillow and fairseq following warnings appear during training start:

No module named 'lightconv_cuda'

If install lightconv-layer from fairseq, the folllowing warning displayed:

WARNING: Unsupported filter length passed - skipping forward pass

Pytorch 1.7
Cuda 10.2
Fairseq 1.0.0a0+19793a7

@idcore I try to install lightconv_layer on both torch 1.7, 1.8 (cuda 11) and they both work. Maybe you should clean your env and try to reinstall it again.

commented

@KinamSalad lightconv_cuda is not a package. It is a pytorch extension write in C++. The shitty thing is fairseq don't automatically install it when you build fairseq(and I don't see and guiding or warning about installing it either) so you have to build it yourself.
You can install it by following steps:

  1. Installing fairseq from source https://github.com/pytorch/fairseq
  2. Go to the folder https://github.com/pytorch/fairseq/tree/master/fairseq/modules/lightconv_layer
  3. Create lightconv_cuda_foward.cu and lightconv_cuda_backward.cu by:
    python cuda_function_gen.py
  4. Install lightconv_layer by:
    python setup.py install

Note: This installation depend a lot on cuda env so.

  • You have to install cuda from Nvidia runtime file or deb file. https://developer.nvidia.com/cuda-downloads
  • If you install cudatookit from pip it will not work(cudatoolkit package is not full cuda installation)
  • If you are in cloud(GCP in my case) you should not use ready build image with cuda. Because they don't install cuda in normal fashion so the code can't find environment variable they need. In my case they prompt $CUDA_HOME variable not found and I can't find where cuda is installed either.
  • I verified it with cuda 11 and torch 1.7, 1.8 on GCP.
commented

Thank you @egochao for your documentation. But someone may still get the WARNING: Unsupported filter length passed - skipping forward pass as @idcore reported, and I resolved it by changing all kernel size of LConvBlock from 17 to 7 (it should be an odd value) since it throws the same warning for the value grater than 7). One thing to notice is that the original paper used 17 filter sizes, so some hyperparameter or model architecture tuning would be required.

commented

Close due to inactivity.

commented

@KinamSalad lightconv_cuda is not a package. It is a pytorch extension write in C++. The shitty thing is fairseq don't automatically install it when you build fairseq(and I don't see and guiding or warning about installing it either) so you have to build it yourself.
You can install it by following steps:

  1. Installing fairseq from source https://github.com/pytorch/fairseq
  2. Go to the folder https://github.com/pytorch/fairseq/tree/master/fairseq/modules/lightconv_layer
  3. Create lightconv_cuda_foward.cu and lightconv_cuda_backward.cu by:
    python cuda_function_gen.py
  4. Install lightconv_layer by:
    python setup.py install

Note: This installation depend a lot on cuda env so.

  • You have to install cuda from Nvidia runtime file or deb file. https://developer.nvidia.com/cuda-downloads
  • If you install cudatookit from pip it will not work(cudatoolkit package is not full cuda installation)
  • If you are in cloud(GCP in my case) you should not use ready build image with cuda. Because they don't install cuda in normal fashion so the code can't find environment variable they need. In my case they prompt $CUDA_HOME variable not found and I can't find where cuda is installed either.
  • I verified it with cuda 11 and torch 1.7, 1.8 on GCP.

So Does it mean I can not train in Conda environment?

Update: I have run it with conda , cudatoolkit 11 and torch 1.8

commented

how to solv it? I have the same problem

commented

how to solv it? I have the same problem

You could check this replacement script:
#11 (comment)

commented

how to solv it? I have the same problem

You could check this replacement script: #11 (comment)

@v-nhandt21 hi, did you solve the install problem of fairseq? If yeah, and did you test whether the replacement script work normal as the lightconv_layer?
I could not install fairseq correctly in my machine, if these two module work the same, I will try repalce this replacement script and wouldn't waste too much time on the installation problems.

Thanks again.

commented

@GuangChen2016 I did not install fairseq, LightweightConv of LEECHOONGHO script worked for me except the soft-DTW of Parallel Tacotron2

ps: The softDTW in this repo can not work with large batch-size. You can check the appendix of this paper, may it help: https://arxiv.org/pdf/2205.04421.pdf

commented

Any concrete luck with no module named 'lightconv_cuda'?