What is _FusedMulAdd of profiling result?
alphaRGB opened this issue · comments
I'm profiling a NLP Transformer model. In the Transformer module, this module contains div mul, but can't find them in profiler result. Only found _FusedMulAdd. I used @tf.function + Eager
Code
matmul_qk = ...
mask = ...
dk = tf.cast(tf.shape(k)[-1], tf.float32)
matmul_qk = matmul_qk / tf.math.sqrt(dk)
matmul_qk += (mask * -1e9)