why dropouts are 0 for codegen-350M-mono?
wasiahmad opened this issue · comments
Wasi Ahmad commented
Hi,
I noticed in the config file (https://huggingface.co/Salesforce/codegen-350M-mono/blob/main/config.json) that:
"attn_pdrop": 0.0
"embd_pdrop": 0.0
"resid_pdrop": 0.0
Is codegen pretrained with dropout 0? @enijkamp
Erik Nijkamp commented
Yes.
In training, the model does not include drop-out regularization, hence dropout of 0.0 in the converted PyTorch forward pass.
Wasi Ahmad commented
@enijkamp What was the reason of not using any dropout? Want to learn if there is any insight. Thanks!