Half precision spectograms - Mixed precision training

[Tom]

Hey,
Is it sufficient to have the spectograms in half precision format (float16)? And did you train your model with mixed precision?

[Yuan Gong]

Hi there,

Is it sufficient to have the spectograms in half precision format (float16)?

I haven't used half-precision before, but I think it is worth having a try.

And did you train your model with mixed precision?

Not for SSAST as it seems torch.cuda.amp doesn't directly support multiple forward functions (e.g., mpg, mpc, etc in https://github.com/YuanGongND/ssast/blob/main/src/models/ast_models.py). But if you can find a workaround, I don't think that hurts the performance, as we use torch.cuda.amp in our original AST project (single forward function), see

https://github.com/YuanGongND/ast/blob/7b2fe7084b622e540643b0d7d7ab736b5eb7683b/src/traintest.py#L117-L122

and

https://github.com/YuanGongND/ast/blob/7b2fe7084b622e540643b0d7d7ab736b5eb7683b/src/models/ast_models.py#L159.