FLOPs/MACs calculation of complicated operations

[hdzhang-code]

Thanks for the code.

I have concern on the accuracy of the FLOPs/MACs calculation of complicated operations such as softmax or relu, while the formula to calculate FLOPS/MACs of simple operation such as linear operation (matrix mulplication) seems fine to me.

For softmax, I see that every single exponential operation is counted as 1 MAC. It is similar when it comes to relu, where every single relu is counted as 1 MAC. However, is the expotential operation implemented really by 1 MAC, i.e. by a mulplication followed by one accumulative adding?

I highly appreciate the work of the project. I am also aware of the difficulty to figure out the CORRECT formula to calculate FLOPs/MACs of every basic operation in PyTorch since some of them are written by C language. However, if this package is of the aforementioned limit (inaccurate FLOPs/MACs counting for complicated operations), being aware of it is helpful for user.

Could you help clarify whether the aformentioned limit exist? Thanks!

[Ligeng Zhu]

The definition of MACs is the total operations of multiplication.

Thus for ReLU, output = max(x, 0), there is no multiplication and the MACs is zero as you can seen in https://github.com/Lyken17/pytorch-OpCounter/blob/master/thop/profile.py

The softmax, as you pointed, it actually depends the actual implementation (e.g., Talyor expandsion, binary search, Newton's method). I here assume the implementation is simplest 1-order taylor. I agree this might be accurate for some platforms, but this usually does not affect much as people do not use softmax in deployment (inference).

[hdzhang-code]

Helpful feedback received. Thanks again for the author's effort.