step time breakdown question
nbcsm opened this issue · comments
In tensorboard, per step time breakdown is displayed.
If I understand correctly, core TF side runs a profiler to capture all kinds of events, e.g. kernel starts/ends, memory H2D, D2H etc.
But given an event, how TF knows which step this event belongs to? Or how TF knows when a step begins/ends?
Tried to dig into TF profiler code, but didn't find a clue...
any comments please?
thanks.
We added step-tracking code to some frameworks such as Keras.
In the upcoming TF 2.3 release, users can also add this step-tracking code
to their own frameworks.
…On Sun, Jun 28, 2020 at 7:33 AM Guoliang Hua ***@***.***> wrote:
In tensorboard, per step time breakdown is displayed.
[image: image]
<https://user-images.githubusercontent.com/32868157/85950079-1578ff80-b98d-11ea-94cf-117c1333078e.png>
If I understand correctly, core TF side runs a profiler to capture all
kinds of events, e.g. kernel starts/ends, memory H2D, D2H etc.
But given an event, how TF knows which step this event belongs to? Or how
TF knows when a step begins/ends?
Tried to dig into TF profiler code, but didn't find a clue...
any comments please?
thanks.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#83>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AE33L3MASDEXCAHFG2YQGH3RY5IDXANCNFSM4OKRZN5A>
.
thanks for the information, @ckluk.
could you help to point out where I can find the "step-tracking" logic please?
really appreciate.