FLOPs/MACs calculation of complicated operations
hdzhang-code opened this issue · comments
Thanks for the code.
I have concern on the accuracy of the FLOPs/MACs calculation of complicated operations such as softmax or relu, while the formula to calculate FLOPS/MACs of simple operation such as linear operation (matrix mulplication) seems fine to me.
For softmax, I see that every single exponential operation is counted as 1 MAC. It is similar when it comes to relu, where every single relu is counted as 1 MAC. However, is the expotential operation implemented really by 1 MAC, i.e. by a mulplication followed by one accumulative adding?
I highly appreciate the work of the project. I am also aware of the difficulty to figure out the CORRECT formula to calculate FLOPs/MACs of every basic operation in PyTorch since some of them are written by C language. However, if this package is of the aforementioned limit (inaccurate FLOPs/MACs counting for complicated operations), being aware of it is helpful for user.
Could you help clarify whether the aformentioned limit exist? Thanks!
The definition of MACs is the total operations of multiplication
.
Thus for ReLU, output = max(x, 0), there is no multiplication and the MACs is zero as you can seen in https://github.com/Lyken17/pytorch-OpCounter/blob/master/thop/profile.py
The softmax, as you pointed, it actually depends the actual implementation (e.g., Talyor expandsion, binary search, Newton's method). I here assume the implementation is simplest 1-order taylor. I agree this might be accurate for some platforms, but this usually does not affect much as people do not use softmax in deployment (inference).
Helpful feedback received. Thanks again for the author's effort.