Giters

tensorflow / profiler

A profiling and performance analysis tool for TensorFlow

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Memory profiling for ops rather than models

zaccharieramzi opened this issue · comments

commented

I am just starting to use the memory profiling which I find super useful. I would love to see some features of memory profiling targeted at smaller operations (rather than model training).

For example, it would be nice to have the ability to zoom in on the y axis as well in the Memory Timeline Graph.

Another example is that the Memory Breakdown Table doesn't feature allocations with size under 1MiB, which could be useful for profiling smaller operations (like a single layer).

commented

Actually I think it's rather a problem of my op not being correctly profiled. Indeed, I handle tensors of size 640 x 368 x 12 (float32), so about 11MB if I am not mistaken, and nothing is logged in the memory breakdown table, except a cryptic "preallocated/unknown" (see screenshot
tf_profiler_wo_memory
).

I will try to provide a reproducible example for memory usage not being logged correctly.

EDIT

I actually think it's because there is something I am missing about the memory breakdown table. I just increased the batch size and the ops of my model are now appearing in it but I don't get what they correspond to.

I think I just need to wait for the documentation. Keeping this open in the meantime.

commented

Actually, if anyone could help me in understanding how to profile correctly the memory I would be super interested. I know this is not necessarily the best place to ask, but I have close to no hope asking on SO for such a recent tool.