Failed to get bounding box

Question

Failed to get bounding box

gaurish1212 opened this issue 9 months ago · comments

Hy

When we build image after change version of mmsc-full==1.7.0 instaed mmcv-full==1.7.2. Then image build successfully and we get status healthy. Then after we hit this command "python image_to_animation.py drawings/garlic.png garlic_out" then we get error.
Traceback (most recent call last):
File "image_to_animation.py", line 41, in
image_to_animation(img_fn, char_anno_dir, motion_cfg_fn, retarget_cfg_fn)
File "image_to_animation.py", line 19, in image_to_animation
image_to_annotations(img_fn, char_anno_dir)
File "C:\Users\DELL4\AnimatedDrawings\examples\image_to_annotations.py", line 53, in image_to_annotations
raise Exception(f"Failed to get bounding box, please check if the 'docker_torchserve' is running and healthy, resp: {resp}")
Exception: Failed to get bounding box, please check if the 'docker_torchserve' is running and healthy, resp: <Response [500]>

lbjhk · Answer 1 · Wed Jan 10 2024 14:15:40 GMT+0800 (China Standard Time)

I guess Torchserve is failing because it doesn't have enough RAM. You may also try to increase RAM for docker.

How about checking your RAM after u enter python image_to_animation.py drawings/garlic.png garlic_out ?

gaurish1212 · Answer 2 · Wed Jan 10 2024 19:52:21 GMT+0800 (China Standard Time)

Hy
How can i increase ram for docker. In my system have 32GB ram

corporateGhost · Answer 3 · Wed Jan 10 2024 20:59:18 GMT+0800 (China Standard Time)

Same Question

lbjhk · Answer 4 · Wed Jan 10 2024 22:44:05 GMT+0800 (China Standard Time)

@gaurish1212 by default, docker can use unlimited RAM in your system.

After running docker run -d --name docker_torchserve -p 8080:8080 -p 8081:8081 docker_torchserve

Try to get more detail from container : docker exec -it docker_torchserve bash

All logs while running docker_torchserve cat /home/torchserve/logs/ts_log.log

Then we may figure out which part goes wrong. Or post it here.

Also I found a simular issue: #136

lbjhk · Answer 5 · Thu Jan 11 2024 02:51:44 GMT+0800 (China Standard Time)

@gaurish1212 try it: #240 (comment)

corporateGhost · Answer 6 · Thu Jan 11 2024 15:15:19 GMT+0800 (China Standard Time)

@gaurish1212 try it: #240 (comment)

I tried this. I changed it from RUN pip install torch to RUN pip install torch==2.0.0
but still it gives me the same error

corporateGhost · Answer 7 · Thu Jan 11 2024 15:42:46 GMT+0800 (China Standard Time)

@gaurish1212 by default, docker can use unlimited RAM in your system.

After running docker run -d --name docker_torchserve -p 8080:8080 -p 8081:8081 docker_torchserve

Try to get more detail from container : docker exec -it docker_torchserve bash

All logs while running docker_torchserve cat /home/torchserve/logs/ts_log.log

Then we may figure out which part goes wrong. Or post it here.

Also I found a simular issue: #136

I tried this and I am getting this
(base) H:\AnimatedDrawings\torchserve>docker exec -it docker_torchserve bash
(base) root@6eab2021b0eb:/# cat /home/torchserve/logs/ts_log.log
cat: /home/torchserve/logs/ts_log.log: No such file or directory

lbjhk · Answer 8 · Thu Jan 18 2024 22:45:48 GMT+0800 (China Standard Time)

@corporateGhost if u can't see ts_log.log , u can try to run bash inside docker container.

step 1:

docker exec -it docker_torchserve bash

step 2:

torchserve --stop

step3:

/opt/conda/bin/torchserve --start --ts-config /home/torchserve/config.properties

Now you can see logs popping up.

gaurish1212 · Answer 9 · Wed Mar 20 2024 20:59:35 GMT+0800 (China Standard Time)

This is our logs
2024-03-20T12:17:46,825 [INFO ] W-9015-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/mmcv/ops/active_rotated_filter.py", line 10, in
2024-03-20T12:17:46,825 [INFO ] W-9015-drawn_humanoid_detector_1.0-stdout MODEL_LOG - ext_module = ext_loader.load_ext(
2024-03-20T12:17:46,825 [INFO ] W-9015-drawn_humanoid_detector_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:17:46,825 [INFO ] W-9015-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/mmcv/utils/ext_loader.py", line 13, in load_ext
2024-03-20T12:17:46,825 [INFO ] W-9015-drawn_humanoid_detector_1.0-stdout MODEL_LOG - ext = importlib.import_module('mmcv.' + name)
2024-03-20T12:17:46,825 [INFO ] W-9015-drawn_humanoid_detector_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:17:46,825 [WARN ] W-9015-drawn_humanoid_detector_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9015-drawn_humanoid_detector_1.0-stderr
2024-03-20T12:17:46,825 [INFO ] W-9015-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/importlib/init.py", line 126, in import_module
2024-03-20T12:17:46,825 [WARN ] W-9015-drawn_humanoid_detector_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9015-drawn_humanoid_detector_1.0-stdout
2024-03-20T12:17:46,825 [INFO ] W-9015-drawn_humanoid_detector_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level)
2024-03-20T12:17:46,825 [INFO ] W-9015-drawn_humanoid_detector_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9015-drawn_humanoid_detector_1.0-stdout
2024-03-20T12:17:46,825 [INFO ] W-9015-drawn_humanoid_detector_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9015 in 55 seconds.
2024-03-20T12:17:46,835 [INFO ] W-9015-drawn_humanoid_detector_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9015-drawn_humanoid_detector_1.0-stderr
2024-03-20T12:17:46,839 [WARN ] W-9023-drawn_humanoid_detector_1.0-stderr MODEL_LOG - /opt/conda/lib/python3.11/site-packages/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
2024-03-20T12:17:46,839 [WARN ] W-9023-drawn_humanoid_detector_1.0-stderr MODEL_LOG - warnings.warn(
2024-03-20T12:17:46,840 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - proceeding without onnxruntime
2024-03-20T12:17:46,840 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - Torch TensorRT not enabled
2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - Backend worker process died.
2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - Traceback (most recent call last):
2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 108, in load
2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler)
2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 153, in _load_handler_file
2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name)
2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/importlib/init.py", line 126, in import_module
2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level)
2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import
2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load
2024-03-20T12:17:46,843 [INFO ] epollEventLoopGroup-5-23 org.pytorch.serve.wlm.WorkerThread - 9023 Worker disconnected. WORKER_STARTED
2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked
2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked
2024-03-20T12:17:46,843 [DEBUG] W-9023-drawn_humanoid_detector_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "", line 940, in exec_module
2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed
2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "/tmp/models/8a762892a49747d4b3d43f92aafb0323/mmdet_handler.py", line 9, in
2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - from mmdet.apis import inference_detector, init_detector
2024-03-20T12:17:46,843 [DEBUG] W-9023-drawn_humanoid_detector_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., responseTimeout:120sec
java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?]
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?]
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?]
at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:229) [model-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/mmdet/apis/init.py", line 2, in
2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - from .inference import (async_inference_detector, inference_detector,
2024-03-20T12:17:46,844 [WARN ] W-9023-drawn_humanoid_detector_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: drawn_humanoid_detector, error: Worker died.
2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/mmdet/apis/inference.py", line 8, in
2024-03-20T12:17:46,844 [DEBUG] W-9023-drawn_humanoid_detector_1.0 org.pytorch.serve.wlm.WorkerThread - W-9023-drawn_humanoid_detector_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - from mmcv.ops import RoIPool
2024-03-20T12:17:46,844 [WARN ] W-9023-drawn_humanoid_detector_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again
2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/mmcv/ops/init.py", line 3, in
2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - from .active_rotated_filter import active_rotated_filter
2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/mmcv/ops/active_rotated_filter.py", line 10, in
2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - ext_module = ext_loader.load_ext(
2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/mmcv/utils/ext_loader.py", line 13, in load_ext
2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - ext = importlib.import_module('mmcv.' + name)
2024-03-20T12:17:46,844 [WARN ] W-9023-drawn_humanoid_detector_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9023-drawn_humanoid_detector_1.0-stderr
2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:17:46,844 [WARN ] W-9023-drawn_humanoid_detector_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9023-drawn_humanoid_detector_1.0-stdout
2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/importlib/init.py", line 126, in import_module
2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9023 in 55 seconds.
2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9023-drawn_humanoid_detector_1.0-stdout
2024-03-20T12:17:46,854 [INFO ] W-9023-drawn_humanoid_detector_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9023-drawn_humanoid_detector_1.0-stderr
2024-03-20T12:18:27,222 [DEBUG] W-9001-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/bin/python, /opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9001, --metrics-config, /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml]
2024-03-20T12:18:27,936 [DEBUG] W-9002-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/bin/python, /opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9002, --metrics-config, /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml]
2024-03-20T12:18:28,050 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9001, pid=14447
2024-03-20T12:18:28,050 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9001
2024-03-20T12:18:28,056 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml.
2024-03-20T12:18:28,057 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - [PID]14447
2024-03-20T12:18:28,057 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Torch worker started.
2024-03-20T12:18:28,057 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Python runtime: 3.11.7
2024-03-20T12:18:28,057 [DEBUG] W-9001-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9001-drawn_humanoid_pose_estimator_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2024-03-20T12:18:28,057 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9001
2024-03-20T12:18:28,058 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9001.
2024-03-20T12:18:28,058 [DEBUG] W-9001-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1710937108058
2024-03-20T12:18:28,058 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1710937108058
2024-03-20T12:18:28,059 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - model_name: drawn_humanoid_pose_estimator, batchSize: 1
2024-03-20T12:18:28,438 [WARN ] W-9001-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - /opt/conda/lib/python3.11/site-packages/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
2024-03-20T12:18:28,438 [WARN ] W-9001-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - warnings.warn(
2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Backend worker process died.
2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Traceback (most recent call last):
2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 263, in
2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - worker.run_server()
2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 231, in run_server
2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket)
2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 194, in handle_connection
2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg)
2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG -
^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:28,439 [INFO ] epollEventLoopGroup-5-6 org.pytorch.serve.wlm.WorkerThread - 9001 Worker disconnected. WORKER_STARTED
2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 131, in load_model
2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service = model_loader.load(
2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 108, in load
2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler)
2024-03-20T12:18:28,439 [DEBUG] W-9001-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG -
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 153, in _load_handler_file
2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name)
2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/importlib/init.py", line 126, in import_module
2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level)
2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import
2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load
2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked
2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked
2024-03-20T12:18:28,439 [DEBUG] W-9001-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., responseTimeout:120sec
java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?]
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?]
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?]
at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:229) [model-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 940, in exec_module
2024-03-20T12:18:28,440 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed
2024-03-20T12:18:28,440 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/tmp/models/34ba3c3d052d4021a508d4689663c503/mmpose_handler.py", line 8, in
2024-03-20T12:18:28,440 [WARN ] W-9001-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: drawn_humanoid_pose_estimator, error: Worker died.
2024-03-20T12:18:28,440 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - from mmpose.apis import (inference_bottom_up_pose_model,
2024-03-20T12:18:28,440 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/mmpose/init.py", line 25, in
2024-03-20T12:18:28,440 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - and mmcv_version <= digit_version(mmcv_maximum_version)),
2024-03-20T12:18:28,440 [DEBUG] W-9001-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9001-drawn_humanoid_pose_estimator_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2024-03-20T12:18:28,440 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:28,440 [WARN ] W-9001-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again
2024-03-20T12:18:28,440 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - AssertionError: MMCV==1.7.2 is used but incompatible. Please install mmcv>=1.3.8, <=1.7.0.
2024-03-20T12:18:28,440 [WARN ] W-9001-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9001-drawn_humanoid_pose_estimator_1.0-stderr
2024-03-20T12:18:28,440 [WARN ] W-9001-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9001-drawn_humanoid_pose_estimator_1.0-stdout
2024-03-20T12:18:28,440 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9001 in 89 seconds.
2024-03-20T12:18:28,449 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9001-drawn_humanoid_pose_estimator_1.0-stderr
2024-03-20T12:18:28,449 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9001-drawn_humanoid_pose_estimator_1.0-stdout
2024-03-20T12:18:28,806 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9002, pid=14461
2024-03-20T12:18:28,806 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9002
2024-03-20T12:18:28,813 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml.
2024-03-20T12:18:28,813 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - [PID]14461
2024-03-20T12:18:28,814 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Torch worker started.
2024-03-20T12:18:28,814 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Python runtime: 3.11.7
2024-03-20T12:18:28,814 [DEBUG] W-9002-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9002-drawn_humanoid_pose_estimator_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2024-03-20T12:18:28,814 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9002
2024-03-20T12:18:28,814 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9002.
2024-03-20T12:18:28,814 [DEBUG] W-9002-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1710937108814
2024-03-20T12:18:28,814 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1710937108814
2024-03-20T12:18:28,815 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - model_name: drawn_humanoid_pose_estimator, batchSize: 1
2024-03-20T12:18:29,154 [WARN ] W-9002-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - /opt/conda/lib/python3.11/site-packages/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
2024-03-20T12:18:29,154 [WARN ] W-9002-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - warnings.warn(
2024-03-20T12:18:29,155 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Backend worker process died.
2024-03-20T12:18:29,155 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Traceback (most recent call last):
2024-03-20T12:18:29,155 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 263, in
2024-03-20T12:18:29,155 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - worker.run_server()
2024-03-20T12:18:29,155 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 231, in run_server
2024-03-20T12:18:29,155 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket)
2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 194, in handle_connection
2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg)
2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG -
^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 131, in load_model
2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service = model_loader.load(
2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:29,156 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9002 Worker disconnected. WORKER_STARTED
2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 108, in load
2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler)
2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG -
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 153, in _load_handler_file
2024-03-20T12:18:29,156 [DEBUG] W-9002-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name)
2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/importlib/init.py", line 126, in import_module
2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level)
2024-03-20T12:18:29,156 [DEBUG] W-9002-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., responseTimeout:120sec
java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?]
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?]
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?]
at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:229) [model-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import
2024-03-20T12:18:29,156 [WARN ] W-9002-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: drawn_humanoid_pose_estimator, error: Worker died.
2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load
2024-03-20T12:18:29,156 [DEBUG] W-9002-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9002-drawn_humanoid_pose_estimator_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked
2024-03-20T12:18:29,156 [WARN ] W-9002-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again
2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked
2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 940, in exec_module
2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed
2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/tmp/models/34ba3c3d052d4021a508d4689663c503/mmpose_handler.py", line 8, in
2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - from mmpose.apis import (inference_bottom_up_pose_model,
2024-03-20T12:18:29,157 [WARN ] W-9002-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9002-drawn_humanoid_pose_estimator_1.0-stderr
2024-03-20T12:18:29,157 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/mmpose/init.py", line 25, in
2024-03-20T12:18:29,157 [WARN ] W-9002-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9002-drawn_humanoid_pose_estimator_1.0-stdout
2024-03-20T12:18:29,157 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - and mmcv_version <= digit_version(mmcv_maximum_version)),
2024-03-20T12:18:29,157 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9002-drawn_humanoid_pose_estimator_1.0-stdout
2024-03-20T12:18:29,157 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9002 in 89 seconds.
2024-03-20T12:18:29,166 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9002-drawn_humanoid_pose_estimator_1.0-stderr
2024-03-20T12:18:29,297 [DEBUG] W-9000-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/bin/python, /opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml]
2024-03-20T12:18:30,115 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=14479
2024-03-20T12:18:30,115 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000
2024-03-20T12:18:30,121 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml.
2024-03-20T12:18:30,121 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - [PID]14479
2024-03-20T12:18:30,121 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Torch worker started.
2024-03-20T12:18:30,121 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Python runtime: 3.11.7
2024-03-20T12:18:30,121 [DEBUG] W-9000-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-drawn_humanoid_pose_estimator_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2024-03-20T12:18:30,121 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000
2024-03-20T12:18:30,122 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000.
2024-03-20T12:18:30,122 [DEBUG] W-9000-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1710937110122
2024-03-20T12:18:30,122 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1710937110122
2024-03-20T12:18:30,122 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - model_name: drawn_humanoid_pose_estimator, batchSize: 1
2024-03-20T12:18:30,465 [WARN ] W-9000-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - /opt/conda/lib/python3.11/site-packages/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
2024-03-20T12:18:30,465 [WARN ] W-9000-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - warnings.warn(
2024-03-20T12:18:30,466 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Backend worker process died.
2024-03-20T12:18:30,466 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Traceback (most recent call last):
2024-03-20T12:18:30,466 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 263, in
2024-03-20T12:18:30,466 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - worker.run_server()
2024-03-20T12:18:30,466 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 231, in run_server
2024-03-20T12:18:30,466 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket)
2024-03-20T12:18:30,466 [INFO ] epollEventLoopGroup-5-2 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED
2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 194, in handle_connection
2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg)
2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG -
^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 131, in load_model
2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service = model_loader.load(
2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 108, in load
2024-03-20T12:18:30,467 [DEBUG] W-9000-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler)
2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG -
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 153, in _load_handler_file
2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name)
2024-03-20T12:18:30,467 [DEBUG] W-9000-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., responseTimeout:120sec
java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?]
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?]
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?]
at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:229) [model-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/importlib/init.py", line 126, in import_module
2024-03-20T12:18:30,467 [WARN ] W-9000-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: drawn_humanoid_pose_estimator, error: Worker died.
2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level)
2024-03-20T12:18:30,467 [DEBUG] W-9000-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-drawn_humanoid_pose_estimator_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:30,467 [WARN ] W-9000-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again
2024-03-20T12:18:30,468 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import
2024-03-20T12:18:30,468 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load
2024-03-20T12:18:30,468 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked
2024-03-20T12:18:30,468 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked
2024-03-20T12:18:30,468 [WARN ] W-9000-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-drawn_humanoid_pose_estimator_1.0-stderr
2024-03-20T12:18:30,468 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 940, in exec_module
2024-03-20T12:18:30,468 [WARN ] W-9000-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-drawn_humanoid_pose_estimator_1.0-stdout
2024-03-20T12:18:30,468 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-drawn_humanoid_pose_estimator_1.0-stdout
2024-03-20T12:18:30,468 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 89 seconds.
2024-03-20T12:18:30,484 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-drawn_humanoid_pose_estimator_1.0-stderr
2024-03-20T12:18:31,949 [DEBUG] W-9010-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/bin/python, /opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9010, --metrics-config, /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml]
2024-03-20T12:18:32,747 [DEBUG] W-9007-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/bin/python, /opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9007, --metrics-config, /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml]
2024-03-20T12:18:32,776 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9010, pid=14495
2024-03-20T12:18:32,776 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9010
2024-03-20T12:18:32,785 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml.
2024-03-20T12:18:32,785 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - [PID]14495
2024-03-20T12:18:32,785 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Torch worker started.
2024-03-20T12:18:32,785 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Python runtime: 3.11.7
2024-03-20T12:18:32,785 [DEBUG] W-9010-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9010-drawn_humanoid_pose_estimator_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2024-03-20T12:18:32,785 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9010
2024-03-20T12:18:32,786 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9010.
2024-03-20T12:18:32,786 [DEBUG] W-9010-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1710937112786
2024-03-20T12:18:32,786 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1710937112786
2024-03-20T12:18:32,786 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - model_name: drawn_humanoid_pose_estimator, batchSize: 1
2024-03-20T12:18:33,163 [WARN ] W-9010-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - /opt/conda/lib/python3.11/site-packages/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
2024-03-20T12:18:33,163 [WARN ] W-9010-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - warnings.warn(
2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Backend worker process died.
2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Traceback (most recent call last):
2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 263, in
2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - worker.run_server()
2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 231, in run_server
2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket)
2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 194, in handle_connection
2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg)
2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG -
^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 131, in load_model
2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service = model_loader.load(
2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 108, in load
2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler)
2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG -
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 153, in _load_handler_file
2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name)
2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/importlib/init.py", line 126, in import_module
2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level)
2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import
2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load
2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked
2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked
2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 940, in exec_module
2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed
2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/tmp/models/34ba3c3d052d4021a508d4689663c503/mmpose_handler.py", line 8, in
2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - from mmpose.apis import (inference_bottom_up_pose_model,
2024-03-20T12:18:33,167 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/mmpose/init.py", line 25, in
2024-03-20T12:18:33,167 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - and mmcv_version <= digit_version(mmcv_maximum_version)),
2024-03-20T12:18:33,167 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:33,167 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - AssertionError: MMCV==1.7.2 is used but incompatible. Please install mmcv>=1.3.8, <=1.7.0.
2024-03-20T12:18:33,167 [INFO ] epollEventLoopGroup-5-3 org.pytorch.serve.wlm.WorkerThread - 9010 Worker disconnected. WORKER_STARTED
2024-03-20T12:18:33,167 [DEBUG] W-9010-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2024-03-20T12:18:33,167 [DEBUG] W-9010-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., responseTimeout:120sec
java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?]
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?]
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?]
at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:229) [model-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2024-03-20T12:18:33,167 [WARN ] W-9010-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: drawn_humanoid_pose_estimator, error: Worker died.
2024-03-20T12:18:33,167 [DEBUG] W-9010-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9010-drawn_humanoid_pose_estimator_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2024-03-20T12:18:33,167 [WARN ] W-9010-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again
2024-03-20T12:18:33,168 [WARN ] W-9010-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9010-drawn_humanoid_pose_estimator_1.0-stderr
2024-03-20T12:18:33,168 [WARN ] W-9010-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9010-drawn_humanoid_pose_estimator_1.0-stdout
2024-03-20T12:18:33,168 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9010 in 89 seconds.
2024-03-20T12:18:33,179 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9010-drawn_humanoid_pose_estimator_1.0-stderr
2024-03-20T12:18:33,180 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9010-drawn_humanoid_pose_estimator_1.0-stdout
2024-03-20T12:18:33,607 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9007, pid=14509
2024-03-20T12:18:33,607 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9007
2024-03-20T12:18:33,614 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml.
2024-03-20T12:18:33,614 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - [PID]14509
2024-03-20T12:18:33,614 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Torch worker started.
2024-03-20T12:18:33,614 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Python runtime: 3.11.7
2024-03-20T12:18:33,614 [DEBUG] W-9007-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9007-drawn_humanoid_pose_estimator_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2024-03-20T12:18:33,614 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9007
2024-03-20T12:18:33,615 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9007.
2024-03-20T12:18:33,615 [DEBUG] W-9007-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1710937113615
2024-03-20T12:18:33,615 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1710937113615
2024-03-20T12:18:33,615 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - model_name: drawn_humanoid_pose_estimator, batchSize: 1
2024-03-20T12:18:33,943 [WARN ] W-9007-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - /opt/conda/lib/python3.11/site-packages/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
2024-03-20T12:18:33,944 [WARN ] W-9007-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - warnings.warn(
2024-03-20T12:18:33,944 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Backend worker process died.
2024-03-20T12:18:33,944 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Traceback (most recent call last):
2024-03-20T12:18:33,944 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 263, in
2024-03-20T12:18:33,944 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - worker.run_server()
2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 231, in run_server
2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket)
2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 194, in handle_connection
2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg)
2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG -
^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:33,945 [INFO ] epollEventLoopGroup-5-4 org.pytorch.serve.wlm.WorkerThread - 9007 Worker disconnected. WORKER_STARTED
2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 131, in load_model
2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service = model_loader.load(
2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 108, in load
2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler)
2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG -
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:33,945 [DEBUG] W-9007-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 153, in _load_handler_file
2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name)
2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/importlib/init.py", line 126, in import_module
2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level)
2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:33,945 [DEBUG] W-9007-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., responseTimeout:120sec
java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?]
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?]
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?]
at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:229) [model-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import
2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load
2024-03-20T12:18:33,945 [WARN ] W-9007-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: drawn_humanoid_pose_estimator, error: Worker died.
2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked
2024-03-20T12:18:33,945 [DEBUG] W-9007-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9007-drawn_humanoid_pose_estimator_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2024-03-20T12:18:33,945 [WARN ] W-9007-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again
2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked
2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 940, in exec_module
2024-03-20T12:18:33,946 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed
2024-03-20T12:18:33,946 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/tmp/models/34ba3c3d052d4021a508d4689663c503/mmpose_handler.py", line 8, in
2024-03-20T12:18:33,946 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - from mmpose.apis import (inference_bottom_up_pose_model,
2024-03-20T12:18:33,946 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/mmpose/init.py", line 25, in
2024-03-20T12:18:33,946 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - and mmcv_version <= digit_version(mmcv_maximum_version)),
2024-03-20T12:18:33,946 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:33,946 [WARN ] W-9007-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9007-drawn_humanoid_pose_estimator_1.0-stderr
2024-03-20T12:18:33,946 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - AssertionError: MMCV==1.7.2 is used but incompatible. Please install mmcv>=1.3.8, <=1.7.0.
2024-03-20T12:18:33,946 [WARN ] W-9007-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9007-drawn_humanoid_pose_estimator_1.0-stdout
2024-03-20T12:18:33,946 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9007-drawn_humanoid_pose_estimator_1.0-stdout
2024-03-20T12:18:33,946 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9007 in 89 seconds.
2024-03-20T12:18:33,955 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9007-drawn_humanoid_pose_estimator_1.0-stderr
2024-03-20T12:18:34,688 [DEBUG] W-9005-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/bin/python, /opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9005, --metrics-config, /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml]
2024-03-20T12:18:34,803 [DEBUG] W-9011-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/bin/python, /opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9011, --metrics-config, /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml]
2024-03-20T12:18:34,852 [DEBUG] W-9006-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/bin/python, /opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9006, --metrics-config, /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml]
2024-03-20T12:18:34,878 [DEBUG] W-9008-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/bin/python, /opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9008, --metrics-config, /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml]
2024-03-20T12:18:34,882 [DEBUG] W-9003-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/bin/python, /opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9003, --metrics-config, /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml]
2024-03-20T12:18:34,889 [DEBUG] W-9009-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/bin/python, /opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9009, --metrics-config, /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml]
2024-03-20T12:18:34,924 [DEBUG] W-9004-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/bin/python, /opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9004, --metrics-config, /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml]
2024-03-20T12:18:35,947 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9005, pid=14527
2024-03-20T12:18:35,947 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9005
2024-03-20T12:18:35,956 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml.
2024-03-20T12:18:35,956 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - [PID]14527
2024-03-20T12:18:35,957 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Torch worker started.
2024-03-20T12:18:35,957 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Python runtime: 3.11.7
2024-03-20T12:18:35,957 [DEBUG] W-9005-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9005-drawn_humanoid_pose_estimator_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2024-03-20T12:18:35,957 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9005
2024-03-20T12:18:35,958 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9005.
2024-03-20T12:18:35,958 [DEBUG] W-9005-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1710937115958
2024-03-20T12:18:35,958 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1710937115958
2024-03-20T12:18:35,958 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - model_name: drawn_humanoid_pose_estimator, batchSize: 1
2024-03-20T12:18:36,172 [INFO ] W-9006-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9006, pid=14533
2024-03-20T12:18:36,172 [INFO ] W-9006-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9006
2024-03-20T12:18:36,173 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9003, pid=14539
2024-03-20T12:18:36,173 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9003
2024-03-20T12:18:36,179 [INFO ] W-9006-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml.
2024-03-20T12:18:36,179 [INFO ] W-9006-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - [PID]14533
2024-03-20T12:18:36,179 [INFO ] W-9006-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Torch worker started.
2024-03-20T12:18:36,179 [INFO ] W-9006-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Python runtime: 3.11.7
2024-03-20T12:18:36,179 [DEBUG] W-9006-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9006-drawn_humanoid_pose_estimator_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2024-03-20T12:18:36,179 [INFO ] W-9006-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9006
2024-03-20T12:18:36,180 [INFO ] W-9006-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9006.
2024-03-20T12:18:36,180 [DEBUG] W-9006-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1710937116180
2024-03-20T12:18:36,180 [INFO ] W-9006-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1710937116180
2024-03-20T12:18:36,181 [INFO ] W-9006-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - model_name: drawn_humanoid_pose_estimator, batchSize: 1
2024-03-20T12:18:36,182 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml.
2024-03-20T12:18:36,182 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - [PID]14539
2024-03-20T12:18:36,182 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Torch worker started.
2024-03-20T12:18:36,182 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Python runtime: 3.11.7
2024-03-20T12:18:36,182 [DEBUG] W-9003-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9003-drawn_humanoid_pose_estimator_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2024-03-20T12:18:36,182 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9003
2024-03-20T12:18:36,183 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9003.
2024-03-20T12:18:36,183 [DEBUG] W-9003-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1710937116183
2024-03-20T12:18:36,183 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1710937116183
2024-03-20T12:18:36,183 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - model_name: drawn_humanoid_pose_estimator, batchSize: 1
2024-03-20T12:18:36,187 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9011, pid=14530
2024-03-20T12:18:36,187 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9011
2024-03-20T12:18:36,194 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml.
2024-03-20T12:18:36,194 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - [PID]14530
2024-03-20T12:18:36,195 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Torch worker started.
2024-03-20T12:18:36,195 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Python runtime: 3.11.7
2024-03-20T12:18:36,195 [DEBUG] W-9011-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9011-drawn_humanoid_pose_estimator_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2024-03-20T12:18:36,195 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9011
2024-03-20T12:18:36,196 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9011.
2024-03-20T12:18:36,196 [DEBUG] W-9011-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1710937116196
2024-03-20T12:18:36,196 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1710937116196
2024-03-20T12:18:36,196 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - model_name: drawn_humanoid_pose_estimator, batchSize: 1
2024-03-20T12:18:36,215 [INFO ] W-9008-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9008, pid=14536
2024-03-20T12:18:36,215 [INFO ] W-9008-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9008
2024-03-20T12:18:36,222 [INFO ] W-9008-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml.
2024-03-20T12:18:36,222 [INFO ] W-9008-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - [PID]14536
2024-03-20T12:18:36,222 [INFO ] W-9008-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Torch worker started.
2024-03-20T12:18:36,222 [INFO ] W-9008-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Python runtime: 3.11.7
2024-03-20T12:18:36,222 [DEBUG] W-9008-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9008-drawn_humanoid_pose_estimator_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2024-03-20T12:18:36,222 [INFO ] W-9008-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9008
2024-03-20T12:18:36,223 [INFO ] W-9008-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9008.
2024-03-20T12:18:36,223 [DEBUG] W-9008-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1710937116223
2024-03-20T12:18:36,223 [INFO ] W-9008-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1710937116223
2024-03-20T12:18:36,224 [INFO ] W-9008-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - model_name: drawn_humanoid_pose_estimator, batchSize: 1
2024-03-20T12:18:36,239 [INFO ] W-9009-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9009, pid=14542
2024-03-20T12:18:36,239 [INFO ] W-9009-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9009
2024-03-20T12:18:36,247 [INFO ] W-9009-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml.
2024-03-20T12:18:36,247 [INFO ] W-9009-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - [PID]14542
2024-03-20T12:18:36,247 [INFO ] W-9009-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Torch worker started.
2024-03-20T12:18:36,247 [INFO ] W-9009-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Python runtime: 3.11.7
2024-03-20T12:18:36,247 [DEBUG] W-9009-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9009-drawn_humanoid_pose_estimator_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2024-03-20T12:18:36,248 [INFO ] W-9009-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9009
2024-03-20T12:18:36,248 [DEBUG] W-9009-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1710937116248
2024-03-20T12:18:36,248 [INFO ] W-9009-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9009.
2024-03-20T12:18:36,249 [INFO ] W-9009-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1710937116249
2024-03-20T12:18:36,249 [INFO ] W-9009-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - model_name: drawn_humanoid_pose_estimator, batchSize: 1
2024-03-20T12:18:36,319 [INFO ] W-9004-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9004, pid=14545
2024-03-20T12:18:36,319 [INFO ] W-9004-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9004
2024-03-20T12:18:36,328 [INFO ] W-9004-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml.
2024-03-20T12:18:36,328 [INFO ] W-9004-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - [PID]14545
2024-03-20T12:18:36,328 [INFO ] W-9004-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Torch worker started.
2024-03-20T12:18:36,328 [INFO ] W-9004-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Python runtime: 3.11.7
2024-03-20T12:18:36,328 [DEBUG] W-9004-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9004-drawn_humanoid_pose_estimator_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2024-03-20T12:18:36,329 [INFO ] W-9004-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9004
2024-03-20T12:18:36,329 [INFO ] W-9004-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9004.
2024-03-20T12:18:36,329 [DEBUG] W-9004-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1710937116329
2024-03-20T12:18:36,329 [INFO ] W-9004-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1710937116329
2024-03-20T12:18:36,330 [INFO ] W-9004-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - model_name: drawn_humanoid_pose_estimator, batchSize: 1
2024-03-20T12:18:36,420 [WARN ] W-9005-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - /opt/conda/lib/python3.11/site-packages/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
2024-03-20T12:18:36,420 [WARN ] W-9005-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - warnings.warn(
2024-03-20T12:18:36,422 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Backend worker process died.
2024-03-20T12:18:36,422 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Traceback (most recent call last):
2024-03-20T12:18:36,422 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 263, in
2024-03-20T12:18:36,422 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - worker.run_server()
2024-03-20T12:18:36,422 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 231, in run_server
2024-03-20T12:18:36,422 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket)
2024-03-20T12:18:36,422 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 194, in handle_connection
2024-03-20T12:18:36,422 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg)
2024-03-20T12:18:36,422 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG -
^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:36,422 [INFO ] epollEventLoopGroup-5-5 org.pytorch.serve.wlm.WorkerThread - 9005 Worker disconnected. WORKER_STARTED
2024-03-20T12:18:36,422 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 131, in load_model
2024-03-20T12:18:36,422 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service = model_loader.load(
2024-03-20T12:18:36,422 [DEBUG] W-9005-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2024-03-20T12:18:36,422 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 108, in load
2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler)
2024-03-20T12:18:36,423 [DEBUG] W-9005-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., responseTimeout:120sec
java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?]
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?]
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?]
at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:229) [model-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2024-03-20T12:18:36,423 [WARN ] W-9005-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: drawn_humanoid_pose_estimator, error: Worker died.
2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG -
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:36,423 [DEBUG] W-9005-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9005-drawn_humanoid_pose_estimator_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 153, in _load_handler_file
2024-03-20T12:18:36,423 [WARN ] W-9005-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again
2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name)
2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/importlib/init.py", line 126, in import_module
2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level)
2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import
2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load
2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked
2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked
2024-03-20T12:18:36,423 [WARN ] W-9005-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9005-drawn_humanoid_pose_estimator_1.0-stderr
2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 940, in exec_module
2024-03-20T12:18:36,424 [WARN ] W-9005-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9005-drawn_humanoid_pose_estimator_1.0-stdout
2024-03-20T12:18:36,424 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9005 in 89 seconds.
2024-03-20T12:18:36,424 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed
2024-03-20T12:18:36,424 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9005-drawn_humanoid_pose_estimator_1.0-stdout
2024-03-20T12:18:36,439 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9005-drawn_humanoid_pose_estimator_1.0-stderr
2024-03-20T12:18:36,642 [WARN ] W-9011-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - /opt/conda/lib/python3.11/site-packages/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
2024-03-20T12:18:36,642 [WARN ] W-9011-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - warnings.warn(
2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Backend worker process died.
2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Traceback (most recent call last):
2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 263, in
2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - worker.run_server()
2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 231, in run_server
2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket)
2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 194, in handle_connection
2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg)
2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG -
^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 131, in load_model
2024-03-20T12:18:36,644 [INFO ] epollEventLoopGroup-5-9 org.pytorch.serve.wlm.WorkerThread - 9011 Worker disconnected. WORKER_STARTED
2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service = model_loader.load(
2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 108, in load
2024-03-20T12:18:36,644 [WARN ] W-9003-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - /opt/conda/lib/python3.11/site-packages/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
2024-03-20T12:18:36,644 [DEBUG] W-9011-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler)
2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG -
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:36,644 [WARN ] W-9003-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - warnings.warn(
2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 153, in _load_handler_file
2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name)
2024-03-20T12:18:36,644 [DEBUG] W-9011-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., responseTimeout:120sec
java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?]
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?]
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?]
at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:229) [model-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:36,644 [WARN ] W-9011-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: drawn_humanoid_pose_estimator, error: Worker died.
2024-03-20T12:18:36,644 [DEBUG] W-9011-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9011-drawn_humanoid_pose_estimator_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2024-03-20T12:18:36,644 [WARN ] W-9011-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again
2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/importlib/init.py", line 126, in import_module
2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level)
2024-03-20T12:18:36,644 [WARN ] W-9006-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - /opt/conda/lib/python3.11/site-packages/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
2024-03-20T12:18:36,645 [WARN ] W-9006-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - warnings.warn(
2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import
2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load
2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked
2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked
2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 940, in exec_module
2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed
2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/tmp/models/34ba3c3d052d4021a508d4689663c503/mmpose_handler.py", line 8, in
2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - from mmpose.apis import (inference_bottom_up_pose_model,
2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/mmpose/init.py", line 25, in
2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - and mmcv_version <= digit_version(mmcv_maximum_version)),
2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - AssertionError: MMCV==1.7.2 is used but incompatible. Please install mmcv>=1.3.8, <=1.7.0.
2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9011-drawn_humanoid_pose_estimator_1.0-stdout
2024-03-20T12:18:36,645 [WARN ] W-9011-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9011-drawn_humanoid_pose_estimator_1.0-stderr
2024-03-20T12:18:36,645 [WARN ] W-9011-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9011-drawn_humanoid_pose_estimator_1.0-stdout
2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9011 in 89 seconds.
2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Backend worker process died.
2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Traceback (most recent call last):
2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 263, in
2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - worker.run_server()
2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 231, in run_server
2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket)
2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 194, in handle_connection
2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg)
2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG -
^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 131, in load_model
2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service = model_loader.load(
2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 108, in load
2024-03-20T12:18:36,646 [INFO ] epollEventLoopGroup-5-8 org.pytorch.serve.wlm.WorkerThread - 9003 Worker disconnected. WORKER_STARTED
2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler)
2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG -
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 153, in _load_handler_file
2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name)
2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/importlib/init.py", line 126, in import_module
2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level)
2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import
2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load
2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked
2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked
2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 940, in exec_module
2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed
2024-03-20T12:18:36,647 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/tmp/models/34ba3c3d052d4021a508d4689663c503/mmpose_handler.py", line 8, in
2024-03-20T12:18:36,647 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - from mmpose.apis import (inference_bottom_up_pose_model,
2024-03-20T12:18:36,647 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/mmpose/init.py", line 25, in
2024-03-20T12:18:36,647 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - and mmcv_version <= digit_version(mmcv_maximum_version)),
2024-03-20T12:18:36,647 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-20T12:18:36,647 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - AssertionError: MMCV==1.7.2 is used but incompatible. Please install mmcv>=1.3.8, <=1.7.0.
2024-03-20T12:18:36,646 [DEBUG] W-9003-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2024-03-20T12:18:36,647 [DEBUG] W-9003-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., responseTimeout:120sec
java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?]
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?]
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?]
at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:229) [model-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2024-03-20T12:18:36,647 [INFO ] W-9006-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Backend worker process died.
2024-03-20T12:18:36,647 [WARN ] W-9003-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: drawn_humanoid_pose_estimator, error: Worker died.
2024-03-20T12:18:36,647 [INFO ] W-9006-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Traceback (most recent call last):
2024-03-20T12:18:36,648 [DEBUG] W-9003-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9003-drawn_humanoid_pose_estimator_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2024-03-20T12:18:36,648 [WARN ] W-9003-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again
2024-03-20T12:18:36,648 [INFO ] W-9006-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 263, in

gaurish1212 · Answer 10 · Wed Mar 20 2024 21:08:24 GMT+0800 (China Standard Time)

We didn't get any output

This is my docker image

Please guide me

gaurish1212 · Answer 11 · Fri Mar 22 2024 13:50:43 GMT+0800 (China Standard Time)

@corporateGhost if u can't see ts_log.log , u can try to run bash inside docker container.

step 1:

docker exec -it docker_torchserve bash

step 2:

torchserve --stop

step3:

/opt/conda/bin/torchserve --start --ts-config /home/torchserve/config.properties

Now you can see logs popping up.

Plaese Reply

gaurish1212 · Answer 12 · Wed Mar 27 2024 22:26:59 GMT+0800 (China Standard Time)

This is my docker logs image what's wrong and how can i fix ?

hjessmith · Answer 13 · Thu Mar 28 2024 04:10:11 GMT+0800 (China Standard Time)

I just merged a PR that deals with the Docker not correctly building. Please try pulling the latest version of the Animated Drawings code and following the installation steps again. Changing mmcv-full version as you did is triggering an assertion error in the torchserve handler code which is causing it to fail.

gaurish1212 · Answer 14 · Wed Apr 24 2024 14:22:33 GMT+0800 (China Standard Time)

solved thanks