trace_view is extremely slow to navigate even with relatively small profiling data

[Cheng Ren]

Hi,
I have been trying tf profiler but trace_view is so slow. I just did profile of 1 step of bert model, and below is the profiling data with their size:

-rw-r--r-- 1 XXX staff 49K Jun 11 23:57 XXX.com.input_pipeline.pb
-rw-r--r-- 1 XXX staff 16M Jun 11 23:57 XXX.com.kernel_stats.pb
-rw-r--r-- 1 XXX staff 2.0G Jun 11 23:57 XXX.memory_profile.pb
-rw-r--r-- 1 XXX staff 51K Jun 11 23:50 XXX.overview_page.pb
-rw-r--r-- 1 XXX staff 7.7M Jun 11 23:54 XXX.tensorflow_stats.pb
-rw-r--r-- 1 XXX staff 17M Jun 11 23:53 XXX.trace.json.gz
-rw-r--r-- 1 XXX staff 1.7G Jun 11 23:58 XXX.xplane.pb

It seems XXX.trace.json.gz is not that big, but how is trace viewer so slow to navigate with such small sized data?

[ckluk]

Hi, If possible, can you send us your profile log directory so that we can reproduce the issue? Thanks, -ck

…

On Mon, Jun 15, 2020 at 2:59 PM Cheng Ren ***@***.***> wrote: Hi, I have been trying tf profiler but trace_view is so slow. I just did profile of 1 step of bert model, and below is the profiling data with their size: -rw-r--r-- 1 XXX staff 49K Jun 11 23:57 XXX.com.input_pipeline.pb -rw-r--r-- 1 XXX staff 16M Jun 11 23:57 XXX.com.kernel_stats.pb -rw-r--r-- 1 XXX staff 2.0G Jun 11 23:57 XXX.memory_profile.pb -rw-r--r-- 1 XXX staff 51K Jun 11 23:50 XXX.overview_page.pb -rw-r--r-- 1 XXX staff 7.7M Jun 11 23:54 XXX.tensorflow_stats.pb -rw-r--r-- 1 XXX staff 17M Jun 11 23:53 XXX.trace.json.gz -rw-r--r-- 1 XXX staff 1.7G Jun 11 23:58 XXX.xplane.pb It seems XXX.trace.json.gz is not that big, but how is trace viewer so slow to navigate with such small sized data? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#65>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE33L3O2VHSF4FUJO2X34TLRW2KULANCNFSM4N634C2A> .

[Cheng Ren]

@ckluk you mean the actual trace.json.gz file?

[ckluk]

Yes.

…

On Tue, Jun 16, 2020 at 8:56 PM Cheng Ren ***@***.***> wrote: @ckluk <https://github.com/ckluk> you mean the actual trace.json.gz file? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#65 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE33L3KJ6JUZKTHJAIEBNRTRXA5HTANCNFSM4N634C2A> .

[Qiumin Xu]

When it is loaded into the brower, we need to unzip and render your trace. Can you try unzip the trace.json.gz and check the size?

[Cheng Ren]

@qiuminxu @ckluk
link to that file. Let me know if it's downloadable for you.
The unzipped file is 292 MB.

[Cheng Ren]

@qiuminxu @ckluk did you find anything?

[Qiumin Xu]

@burgerkingeater Thanks for the reproducer. I can replicate on my side, but haven't root cause yet.
On the other hand, you can upload the trace.json.gz file to chrom://tracing while we're debugging this issue.

[Qiumin Xu]

@burgerkingeater The nightly version includes a change that improves the speed. Can you try the nightly version of the profiler? (https://pypi.org/project/tbp-nightly/)
pip3 install -U tbp-nightly

[Cheng Ren]

Thanks for the investigation. I will try that, in the meantime, can you point me to the PR(s) which improves the speed?

…

On Mon, Jun 22, 2020 at 1:52 PM Qiumin Xu ***@***.***> wrote: @burgerkingeater <https://github.com/burgerkingeater> The nightly version includes a change that improves the speed. Can you try the nightly version of the profiler? (https://pypi.org/project/tbp-nightly/) pip3 install -U tbp-nightly — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#65 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAK4WZ5BNFNMUSISRW4IVUDRX7AA7ANCNFSM4N634C2A> .

[Qiumin Xu]

There isn't a PR in this repo. Only the trace viewer version is updated:
https://github.com/tensorflow/profiler/blob/master/plugin/third_party/tracing/BUILD#L16

[Cheng Ren]

@qiuminxu which tb version could it be compatible with?
i encountered following error with tb nightly :
(base) ➜ 10w10g_2 tensorboard --logdir .
Traceback (most recent call last):
File "/usr/local/anaconda3/bin/tensorboard", line 8, in
sys.exit(run_main())
File "/usr/local/anaconda3/lib/python3.7/site-packages/tensorboard/main.py", line 65, in run_main
default.get_plugins(),
File "/usr/local/anaconda3/lib/python3.7/site-packages/tensorboard/default.py", line 101, in get_plugins
return get_static_plugins() + get_dynamic_plugins()
File "/usr/local/anaconda3/lib/python3.7/site-packages/tensorboard/default.py", line 139, in get_dynamic_plugins
"tensorboard_plugins"
File "/usr/local/anaconda3/lib/python3.7/site-packages/tensorboard/default.py", line 138, in
for entry_point in pkg_resources.iter_entry_points(
File "/usr/local/anaconda3/lib/python3.7/site-packages/pkg_resources/init.py", line 2446, in load
return self.resolve()
File "/usr/local/anaconda3/lib/python3.7/site-packages/pkg_resources/init.py", line 2452, in resolve
module = import(self.module_name, fromlist=['name'], level=0)
ModuleNotFoundError: No module named 'tensorboard_plugin_profile.profile_plugin_loader'

[Qiumin Xu]

Have you installed tbp-nightly? It is compatible with TB 2.2+.

Alternatively, you can try https://github.com/tensorflow/profiler#quick-start
This will install a nightly version of the profiler in a clean virtual environment.

[Cheng Ren]

hi @qiuminxu the nightly one is faster, thanks. when you said you have a repro, you mean you have reproduced it on 2.2?

[Qiumin Xu]

That's good to know. Yes, I was reproducing in 2.2.