__new__() got an unexpected keyword argument 'delay_ms' when testing inference request
rbavery opened this issue · comments
The Environment
I tried using tensorboard, tensorboard plugin and tensorflow installations outside of the virtual env suggested in the README, they satisfy the versions specified but produce an error.
tensorboard: '2.4.0'
tensorflow: '2.3.1'
tensorboard_plugin_profile: 2.4
The Error
I'm running the tutorial line by line here, but I cannot capture any profile data. When I try to, I get a notice in tensorboard that says
Failed to capture profile: [object Object]
And in my terminal window where I'm running tensorboard, this shows up when I request:
tensorboard --logdir ~/logs/inference_demo/ --port 6006
2021-01-12 15:13:15.239907: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all
TensorBoard 2.4.0 at http://localhost:6006/ (Press CTRL+C to quit)
W0112 15:13:47.703725 140358682797824 security_validator.py:51] In 3.0, this warning will become an error:
Illegal Content-Security-Policy for script-src: 'unsafe-inline'
E0112 15:14:02.757672 140358682797824 _internal.py:113] Error on request:
Traceback (most recent call last):
File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/werkzeug/serving.py", line 323, in run_wsgi
execute(self.server.app)
File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/werkzeug/serving.py", line 312, in execute
application_iter = app(environ, start_response)
File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/application.py", line 526, in __call__
return self._app(environ, start_response)
File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/application.py", line 567, in wrapper
return wsgi_app(environ, start_response)
File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/security_validator.py", line 82, in __call__
return self._application(environ, start_response_proxy)
File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/path_prefix.py", line 71, in __call__
return self._application(environ, start_response)
File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/experiment_id.py", line 76, in __call__
return self._application(environ, start_response)
File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/empty_path_redirect.py", line 47, in __call__
return self._application(environ, start_response)
File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/application.py", line 590, in wrapper
return wsgi_app(environ, start_response)
File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/application.py", line 549, in _route_request
return self.exact_routes[clean_path](environ, start_response)
File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/werkzeug/wrappers/base_request.py", line 238, in application
resp = f(*args[:-2] + (request,))
File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard_plugin_profile/profile_plugin.py", line 526, in capture_route
delay_ms=int(request.args.get('delay', '0'))
TypeError: __new__() got an unexpected keyword argument 'delay_ms'
E0112 15:14:55.211221 140358682797824 _internal.py:113] Error on request:
Traceback (most recent call last):
File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/werkzeug/serving.py", line 323, in run_wsgi
execute(self.server.app)
File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/werkzeug/serving.py", line 312, in execute
application_iter = app(environ, start_response)
File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/application.py", line 526, in __call__
return self._app(environ, start_response)
File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/application.py", line 567, in wrapper
return wsgi_app(environ, start_response)
File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/security_validator.py", line 82, in __call__
return self._application(environ, start_response_proxy)
File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/path_prefix.py", line 71, in __call__
return self._application(environ, start_response)
File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/experiment_id.py", line 76, in __call__
return self._application(environ, start_response)
File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/empty_path_redirect.py", line 47, in __call__
return self._application(environ, start_response)
File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/application.py", line 590, in wrapper
return wsgi_app(environ, start_response)
File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard/backend/application.py", line 549, in _route_request
return self.exact_routes[clean_path](environ, start_response)
File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/werkzeug/wrappers/base_request.py", line 238, in application
resp = f(*args[:-2] + (request,))
File "/home/rave/miniconda3/envs/data-science-stack-2.5.0/lib/python3.7/site-packages/tensorboard_plugin_profile/profile_plugin.py", line 526, in capture_route
delay_ms=int(request.args.get('delay', '0'))
TypeError: __new__() got an unexpected keyword argument 'delay_ms'
What I've tried
From the instructions:
Press Capture on the TensorBoard UI and send an inference request to TF Serving quickly thereafter.
I've tried to send the post request to the example tfserving container "quickly" following the instructions, but no matter how quickly I do so, it seems like the unexpected keyword is the problem.
It seems like there is a version conflict between tensorflow 's profiler and the plugin, so perhaps the suggested prerequisites can be updated.
I have exactly the same problem. Did you find the solution?
This is because 'delay_ms' is a new parameter that we added in TensorFlow 2.4 and set in the profiler plugin. A fix is on the way. Once it lands you can use a nightly release of TensorBoard profiler plugin[1]. Otherwise, you can also upgrade to TensorFlow 2.4.
Thanks for the helpful update @yisitu
There is still a chance that you have to update to TensorFlow 2.4 to remedy the issue, but we can give the quick fix a try first.
Could you give this tbp-nightly a try? https://pypi.org/project/tbp-nightly/2.5.0a20210217/#modal-close
I don't think I'm getting this issue anymore, I'm only using the profiler/install_and_run.py script to set up my profiler instance now.
2021-02-18 16:14:28.085304: I tensorflow/core/profiler/rpc/client/capture_profile.cc:198] Profiler delay_ms was 0, start_timestamp_ns set to 1613693668085285586 [2021-02-18T16:14:28.085285586-08:00]