LightWeightConv layer warnings during training
idcore opened this issue · comments
If just install specified requirements + Pillow and fairseq following warnings appear during training start:
No module named 'lightconv_cuda'
If install lightconv-layer from fairseq, the folllowing warning displayed:
WARNING: Unsupported filter length passed - skipping forward pass
Pytorch 1.7
Cuda 10.2
Fairseq 1.0.0a0+19793a7
@idcore
I also have a same problem.
Just installing fairseq does not solve this problem.
How were you able to install lightconv_layer independently?
If just install specified requirements + Pillow and fairseq following warnings appear during training start:
No module named 'lightconv_cuda'
If install lightconv-layer from fairseq, the folllowing warning displayed:
WARNING: Unsupported filter length passed - skipping forward pass
Pytorch 1.7
Cuda 10.2
Fairseq 1.0.0a0+19793a7
@idcore I try to install lightconv_layer on both torch 1.7, 1.8 (cuda 11) and they both work. Maybe you should clean your env and try to reinstall it again.
@KinamSalad lightconv_cuda is not a package. It is a pytorch extension write in C++. The shitty thing is fairseq don't automatically install it when you build fairseq(and I don't see and guiding or warning about installing it either) so you have to build it yourself.
You can install it by following steps:
- Installing fairseq from source https://github.com/pytorch/fairseq
- Go to the folder https://github.com/pytorch/fairseq/tree/master/fairseq/modules/lightconv_layer
- Create lightconv_cuda_foward.cu and lightconv_cuda_backward.cu by:
python cuda_function_gen.py
- Install lightconv_layer by:
python setup.py install
Note: This installation depend a lot on cuda env so.
- You have to install cuda from Nvidia runtime file or deb file. https://developer.nvidia.com/cuda-downloads
- If you install cudatookit from pip it will not work(cudatoolkit package is not full cuda installation)
- If you are in cloud(GCP in my case) you should not use ready build image with cuda. Because they don't install cuda in normal fashion so the code can't find environment variable they need. In my case they prompt $CUDA_HOME variable not found and I can't find where cuda is installed either.
- I verified it with cuda 11 and torch 1.7, 1.8 on GCP.
Thank you @egochao for your documentation. But someone may still get the WARNING: Unsupported filter length passed - skipping forward pass
as @idcore reported, and I resolved it by changing all kernel size of LConvBlock
from 17
to 7
(it should be an odd value) since it throws the same warning for the value grater than 7
). One thing to notice is that the original paper used 17
filter sizes, so some hyperparameter or model architecture tuning would be required.
Close due to inactivity.
@KinamSalad lightconv_cuda is not a package. It is a pytorch extension write in C++. The shitty thing is fairseq don't automatically install it when you build fairseq(and I don't see and guiding or warning about installing it either) so you have to build it yourself.
You can install it by following steps:
- Installing fairseq from source https://github.com/pytorch/fairseq
- Go to the folder https://github.com/pytorch/fairseq/tree/master/fairseq/modules/lightconv_layer
- Create lightconv_cuda_foward.cu and lightconv_cuda_backward.cu by:
python cuda_function_gen.py
- Install lightconv_layer by:
python setup.py install
Note: This installation depend a lot on cuda env so.
- You have to install cuda from Nvidia runtime file or deb file. https://developer.nvidia.com/cuda-downloads
- If you install cudatookit from pip it will not work(cudatoolkit package is not full cuda installation)
- If you are in cloud(GCP in my case) you should not use ready build image with cuda. Because they don't install cuda in normal fashion so the code can't find environment variable they need. In my case they prompt $CUDA_HOME variable not found and I can't find where cuda is installed either.
- I verified it with cuda 11 and torch 1.7, 1.8 on GCP.
So Does it mean I can not train in Conda environment?
Update: I have run it with conda , cudatoolkit 11 and torch 1.8
how to solv it? I have the same problem
how to solv it? I have the same problem
You could check this replacement script:
#11 (comment)
how to solv it? I have the same problem
You could check this replacement script: #11 (comment)
@v-nhandt21 hi, did you solve the install problem of fairseq? If yeah, and did you test whether the replacement script work normal as the lightconv_layer?
I could not install fairseq correctly in my machine, if these two module work the same, I will try repalce this replacement script and wouldn't waste too much time on the installation problems.
Thanks again.
@GuangChen2016 I did not install fairseq, LightweightConv of LEECHOONGHO script worked for me except the soft-DTW of Parallel Tacotron2
ps: The softDTW in this repo can not work with large batch-size. You can check the appendix of this paper, may it help: https://arxiv.org/pdf/2205.04421.pdf
Any concrete luck with no module named 'lightconv_cuda'
?