how does tf profiler work with estimator api

Question

how does tf profiler work with estimator api

burgerkingeater opened this issue 4 years ago · comments

Is the following way the right usage to enable profiler with tf estimator?

    tf.profiler.experimental.start(FLAGS.working_dir)
    tf.estimator.train_and_evaluate(
        classifier,
        train_spec=tf.estimator.TrainSpec(input_fn=input_fn, max_steps=FLAGS.steps, hooks=hooks),
        eval_spec=tf.estimator.EvalSpec(input_fn=input_fn)
    )
    tf.profiler.experimental.stop()

However the profiling result doesn't include any steps:

Jiho Choi · Answer 1 · Thu Jul 02 2020 07:20:19 GMT+0800 (China Standard Time)

It should work with tf.estimator. Can you share your log files?

Cheng Ren · Answer 2 · Thu Jul 02 2020 07:57:46 GMT+0800 (China Standard Time)

@jray319 does the profiler setup code make sense to you?

Jiho Choi · Answer 3 · Thu Jul 02 2020 09:32:07 GMT+0800 (China Standard Time)

tf.profiler.experimental.stop doesn't accept the log directory path.

https://www.tensorflow.org/api_docs/python/tf/profiler/experimental/stop

Other than that, it looks fine.

Jiho Choi · Answer 4 · Thu Jul 02 2020 10:50:19 GMT+0800 (China Standard Time)

And can you confirm if you see anything in the trace viewer first? If so, can you also check if you see FunctionRun or SessionRun events? A screenshot of the trace viewer would help too.

Cheng Ren · Answer 5 · Thu Jul 23 2020 06:55:14 GMT+0800 (China Standard Time)

@jray319 log file:

(tf-edge-azkaban)[chren@chren-ld2 tf-edge_trunk]$ python ./tf-edge-azkaban/src/linkedin/tensorflow/mnist/mnist_estimator.py --notb --download_data True --steps 50
2020-07-22 15:52:29.945562: W tensorflow_io/core/kernels/audio_video_mp3_kernels.cc:252] libmp3lame.so.0 or lame functions are not available
2020-07-22 15:52:29.946027: I tensorflow_io/core/kernels/cpu_check.cc:128] Your CPU supports instructions that this TensorFlow IO binary was not compiled to use: SSE3 SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
I0722 15:52:30.071607 140031036004160 mnist_estimator.py:76] Using single node training strategy
INFO:tensorflow:Using config: {'_model_dir': '/tmp/tensorflow/mnist/working_dir', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
I0722 15:52:30.120822 140031036004160 estimator.py:212] Using config: {'_model_dir': '/tmp/tensorflow/mnist/working_dir', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
2020-07-22 15:52:30.121254: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started.
2020-07-22 15:52:30.122642: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-07-22 15:52:30.597918: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1363] Profiler found 1 GPUs
2020-07-22 15:52:30.598326: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcupti.so.10.1'; dlerror: libcupti.so.10.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /export/apps/xtools/oracle-instantclient
2020-07-22 15:52:30.598371: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1408] function cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error CUPTI could not be loaded or symbol could not be found.
2020-07-22 15:52:30.598403: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1447] function cupti_interface_->ActivityRegisterCallbacks( AllocCuptiActivityBuffer, FreeCuptiActivityBuffer)failed with error CUPTI could not be loaded or symbol could not be found.
INFO:tensorflow:Not using Distribute Coordinator.
I0722 15:52:30.598795 140031036004160 estimator_training.py:186] Not using Distribute Coordinator.
INFO:tensorflow:Running training and evaluation locally (non-distributed).
I0722 15:52:30.599198 140031036004160 training.py:612] Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.
I0722 15:52:30.599605 140031036004160 training.py:700] Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.
WARNING:tensorflow:From /home/chren/work/gitli/tf-edge_trunk/build/tf-edge-azkaban/venv/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py:1666: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
W0722 15:52:30.608601 140031036004160 deprecation.py:506] From /home/chren/work/gitli/tf-edge_trunk/build/tf-edge-azkaban/venv/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py:1666: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
WARNING:tensorflow:From /home/chren/work/gitli/tf-edge_trunk/build/tf-edge-azkaban/venv/lib/python3.7/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
W0722 15:52:30.609014 140031036004160 deprecation.py:323] From /home/chren/work/gitli/tf-edge_trunk/build/tf-edge-azkaban/venv/lib/python3.7/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
2020-07-22 15:52:30.616995: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:21:00.0 name: Quadro P400 computeCapability: 6.1
coreClock: 1.2525GHz coreCount: 2 deviceMemorySize: 1.95GiB deviceMemoryBandwidth: 29.88GiB/s
2020-07-22 15:52:30.617466: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-07-22 15:52:30.619688: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-07-22 15:52:30.622152: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-07-22 15:52:30.622660: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-07-22 15:52:30.625414: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-07-22 15:52:30.626862: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-07-22 15:52:30.627110: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudnn.so.7'; dlerror: libcudnn.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /export/apps/xtools/oracle-instantclient
2020-07-22 15:52:30.627133: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1598] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
I0722 15:52:30.641401 140031036004160 dataset_builder.py:199] Overwrite dataset info from restored data version.
I0722 15:52:30.643471 140031036004160 dataset_info.py:403] Field info.location from disk and from code do not match. Keeping the one from code.
I0722 15:52:30.644027 140031036004160 dataset_builder.py:285] Reusing dataset mnist (/tmp/tensorflow/mnist/input_data/mnist/3.0.0)
I0722 15:52:30.644193 140031036004160 dataset_builder.py:458] Constructing tf.data.Dataset for split None, from /tmp/tensorflow/mnist/input_data/mnist/3.0.0
INFO:tensorflow:Calling model_fn.
I0722 15:52:31.027919 140031036004160 estimator.py:1147] Calling model_fn.
INFO:tensorflow:Done calling model_fn.
I0722 15:52:31.237249 140031036004160 estimator.py:1149] Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
I0722 15:52:31.238812 140031036004160 basic_session_run_hooks.py:546] Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
I0722 15:52:32.042130 140031036004160 monitored_session.py:246] Graph was finalized.
2020-07-22 15:52:32.042580: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2020-07-22 15:52:32.053413: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 1800000000 Hz
2020-07-22 15:52:32.054763: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x45f2c00 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-07-22 15:52:32.054812: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-07-22 15:52:32.137509: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x45f5870 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-07-22 15:52:32.137640: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Quadro P400, Compute Capability 6.1
2020-07-22 15:52:32.139024: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:21:00.0 name: Quadro P400 computeCapability: 6.1
coreClock: 1.2525GHz coreCount: 2 deviceMemorySize: 1.95GiB deviceMemoryBandwidth: 29.88GiB/s
2020-07-22 15:52:32.139172: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-07-22 15:52:32.139256: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-07-22 15:52:32.139331: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-07-22 15:52:32.139411: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-07-22 15:52:32.139470: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-07-22 15:52:32.139541: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-07-22 15:52:32.140201: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudnn.so.7'; dlerror: libcudnn.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /export/apps/xtools/oracle-instantclient
2020-07-22 15:52:32.140282: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1598] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2020-07-22 15:52:32.140414: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-07-22 15:52:32.140465: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0
2020-07-22 15:52:32.140522: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N
INFO:tensorflow:Running local_init_op.
I0722 15:52:32.224800 140031036004160 session_manager.py:505] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0722 15:52:32.235003 140031036004160 session_manager.py:508] Done running local_init_op.
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
I0722 15:52:33.507599 140031036004160 basic_session_run_hooks.py:614] Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Saving checkpoints for 0 into /tmp/tensorflow/mnist/working_dir/model.ckpt.
I0722 15:52:33.508706 140031036004160 basic_session_run_hooks.py:618] Saving checkpoints for 0 into /tmp/tensorflow/mnist/working_dir/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
I0722 15:52:34.088637 140031036004160 basic_session_run_hooks.py:626] Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:loss = 2.2899237, step = 0
I0722 15:52:35.013373 140031036004160 basic_session_run_hooks.py:262] loss = 2.2899237, step = 0
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 50...
I0722 15:52:35.691913 140031036004160 basic_session_run_hooks.py:614] Calling checkpoint listeners before saving checkpoint 50...
INFO:tensorflow:Saving checkpoints for 50 into /tmp/tensorflow/mnist/working_dir/model.ckpt.
I0722 15:52:35.692324 140031036004160 basic_session_run_hooks.py:618] Saving checkpoints for 50 into /tmp/tensorflow/mnist/working_dir/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 50...
I0722 15:52:36.238993 140031036004160 basic_session_run_hooks.py:626] Calling checkpoint listeners after saving checkpoint 50...
I0722 15:52:36.248476 140031036004160 dataset_builder.py:199] Overwrite dataset info from restored data version.
I0722 15:52:36.250970 140031036004160 dataset_info.py:403] Field info.location from disk and from code do not match. Keeping the one from code.
I0722 15:52:36.251540 140031036004160 dataset_builder.py:285] Reusing dataset mnist (/tmp/tensorflow/mnist/input_data/mnist/3.0.0)
I0722 15:52:36.251745 140031036004160 dataset_builder.py:458] Constructing tf.data.Dataset for split None, from /tmp/tensorflow/mnist/input_data/mnist/3.0.0
INFO:tensorflow:Calling model_fn.
I0722 15:52:36.497907 140031036004160 estimator.py:1147] Calling model_fn.
INFO:tensorflow:Done calling model_fn.
I0722 15:52:36.570430 140031036004160 estimator.py:1149] Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2020-07-22T15:52:36Z
I0722 15:52:36.592642 140031036004160 evaluation.py:255] Starting evaluation at 2020-07-22T15:52:36Z
INFO:tensorflow:Graph was finalized.
I0722 15:52:36.654996 140031036004160 monitored_session.py:246] Graph was finalized.
2020-07-22 15:52:36.656286: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:21:00.0 name: Quadro P400 computeCapability: 6.1
coreClock: 1.2525GHz coreCount: 2 deviceMemorySize: 1.95GiB deviceMemoryBandwidth: 29.88GiB/s
2020-07-22 15:52:36.656381: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-07-22 15:52:36.656408: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-07-22 15:52:36.656455: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-07-22 15:52:36.656478: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-07-22 15:52:36.656501: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-07-22 15:52:36.656537: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-07-22 15:52:36.656820: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudnn.so.7'; dlerror: libcudnn.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /export/apps/xtools/oracle-instantclient
2020-07-22 15:52:36.656848: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1598] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2020-07-22 15:52:36.656886: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-07-22 15:52:36.656905: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0
2020-07-22 15:52:36.656923: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N
INFO:tensorflow:Restoring parameters from /tmp/tensorflow/mnist/working_dir/model.ckpt-50
I0722 15:52:36.658340 140031036004160 saver.py:1293] Restoring parameters from /tmp/tensorflow/mnist/working_dir/model.ckpt-50
INFO:tensorflow:Running local_init_op.
I0722 15:52:36.700282 140031036004160 session_manager.py:505] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0722 15:52:36.709971 140031036004160 session_manager.py:508] Done running local_init_op.
INFO:tensorflow:Evaluation [10/100]
I0722 15:52:37.338090 140031036004160 evaluation.py:167] Evaluation [10/100]
INFO:tensorflow:Evaluation [20/100]
I0722 15:52:37.395689 140031036004160 evaluation.py:167] Evaluation [20/100]
INFO:tensorflow:Evaluation [30/100]
I0722 15:52:37.449104 140031036004160 evaluation.py:167] Evaluation [30/100]
INFO:tensorflow:Evaluation [40/100]
I0722 15:52:37.504137 140031036004160 evaluation.py:167] Evaluation [40/100]
INFO:tensorflow:Evaluation [50/100]
I0722 15:52:37.563459 140031036004160 evaluation.py:167] Evaluation [50/100]
INFO:tensorflow:Evaluation [60/100]
I0722 15:52:37.622996 140031036004160 evaluation.py:167] Evaluation [60/100]
INFO:tensorflow:Evaluation [70/100]
I0722 15:52:37.683862 140031036004160 evaluation.py:167] Evaluation [70/100]
INFO:tensorflow:Evaluation [80/100]
I0722 15:52:37.743713 140031036004160 evaluation.py:167] Evaluation [80/100]
INFO:tensorflow:Evaluation [90/100]
I0722 15:52:37.801715 140031036004160 evaluation.py:167] Evaluation [90/100]
INFO:tensorflow:Evaluation [100/100]
I0722 15:52:37.860366 140031036004160 evaluation.py:167] Evaluation [100/100]
INFO:tensorflow:Inference Time : 1.34750s
I0722 15:52:37.940371 140031036004160 evaluation.py:273] Inference Time : 1.34750s
INFO:tensorflow:Finished evaluation at 2020-07-22-15:52:37
I0722 15:52:37.943020 140031036004160 evaluation.py:276] Finished evaluation at 2020-07-22-15:52:37
INFO:tensorflow:Saving dict for global step 50: global_step = 50, loss = 1.7582098
I0722 15:52:37.943607 140031036004160 estimator.py:2049] Saving dict for global step 50: global_step = 50, loss = 1.7582098
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 50: /tmp/tensorflow/mnist/working_dir/model.ckpt-50
I0722 15:52:38.662950 140031036004160 estimator.py:2109] Saving 'checkpoint_path' summary for global step 50: /tmp/tensorflow/mnist/working_dir/model.ckpt-50
2020-07-22 15:52:38.670383: W tensorflow/core/kernels/data/cache_dataset_ops.cc:794] The calling iterator did not fully read the dataset being cached. In order to avoid unexpected truncation of the dataset, the partially cached contents of the dataset will be discarded. This can happen if you have an input pipeline similar to `dataset.cache().take(k).repeat()`. You should use `dataset.take(k).cache().repeat()` instead.
INFO:tensorflow:Loss for final step: 1.7429798.
I0722 15:52:38.734550 140031036004160 estimator.py:371] Loss for final step: 1.7429798.
2020-07-22 15:52:38.736590: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1430] function cupti_interface_->EnableCallback( 0 , subscriber_, CUPTI_CB_DOMAIN_DRIVER_API, cbid)failed with error CUPTI could not be loaded or symbol could not be found.
2020-07-22 15:52:39.968276: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216]  GpuTracer has collected 0 callback api events and 0 activity events.
2020-07-22 15:52:43.207412: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /tmp/tensorflow/mnist/working_dir/plugins/profile/2020_07_22_15_52_40
2020-07-22 15:52:45.932315: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /tmp/tensorflow/mnist/working_dir/plugins/profile/2020_07_22_15_52_40/chren-ld2.trace.json.gz
2020-07-22 15:52:47.017829: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 0.002 ms

2020-07-22 15:52:47.020657: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /tmp/tensorflow/mnist/working_dir/plugins/profile/2020_07_22_15_52_40Dumped tool data for overview_page.pb to /tmp/tensorflow/mnist/working_dir/plugins/profile/2020_07_22_15_52_40/chren-ld2.overview_page.pb
Dumped tool data for input_pipeline.pb to /tmp/tensorflow/mnist/working_dir/plugins/profile/2020_07_22_15_52_40/chren-ld2.input_pipeline.pb
Dumped tool data for tensorflow_stats.pb to /tmp/tensorflow/mnist/working_dir/plugins/profile/2020_07_22_15_52_40/chren-ld2.tensorflow_stats.pb
Dumped tool data for kernel_stats.pb to /tmp/tensorflow/mnist/working_dir/plugins/profile/2020_07_22_15_52_40/chren-ld2.kernel_stats.pb

Cheng Ren · Answer 6 · Thu Jul 23 2020 06:58:47 GMT+0800 (China Standard Time)

i don't see functionrun in trace viewer:

i do see sessionrun:

Cheng Ren · Answer 7 · Sat Jul 25 2020 02:48:12 GMT+0800 (China Standard Time)

@jray319 would you mind taking a look?

Keqiu Hu · Answer 8 · Mon Mar 15 2021 07:26:08 GMT+0800 (China Standard Time)

same issue.. does profiler work with estimator?