chaos-tproxy-controller cannot graceful shut down
Hexilee opened this issue · comments
After chaos-mesh/chaos-mesh#2668, we have the graceful shutdown in Chaos Mesh. However, I found the controller of chaos-tproxy cannot be killed by SIGINT or SIGTERM.
root@chaos-daemon-nc2p5:/# ps aux | grep tproxy
root 1476116 0.0 0.0 3388 2496 ? S 17:13 0:00 /usr/local/bin/nsexec -l -n /proc/1469150/ns/net -p /proc/1469150/ns/pid -- /usr/local/bin/tproxy -i -vv
root 1476117 0.0 0.0 549552 6348 ? Sl 17:13 0:00 /usr/local/bin/tproxy -i -vv
root 1476779 0.0 0.0 3084 888 pts/0 R+ 17:14 0:00 grep tproxy
root@chaos-daemon-nc2p5:/# kill 1476117
root@chaos-daemon-nc2p5:/# ps aux | grep tproxy
root 1476116 0.0 0.0 3388 2496 ? S 17:13 0:00 /usr/local/bin/nsexec -l -n /proc/1469150/ns/net -p /proc/1469150/ns/pid -- /usr/local/bin/tproxy -i -vv
root 1476117 0.0 0.0 549552 6348 ? Sl 17:13 0:00 /usr/local/bin/tproxy -i -vv
root 1476805 0.0 0.0 3084 892 pts/0 S+ 17:14 0:00 grep tproxy
root@chaos-daemon-nc2p5:/# ps aux | grep tproxy
root 1476116 0.0 0.0 3388 2496 ? S 17:13 0:00 /usr/local/bin/nsexec -l -n /proc/1469150/ns/net -p /proc/1469150/ns/pid -- /usr/local/bin/tproxy -i -vv
root 1476117 0.0 0.0 549552 6348 ? Sl 17:13 0:00 /usr/local/bin/tproxy -i -vv
root 1476807 0.0 0.0 3084 888 pts/0 S+ 17:14 0:00 grep tproxy
root@chaos-daemon-nc2p5:/# kill -15 1476117
root@chaos-daemon-nc2p5:/# ps aux | grep tproxy
root 1476116 0.0 0.0 3388 2496 ? S 17:13 0:00 /usr/local/bin/nsexec -l -n /proc/1469150/ns/net -p /proc/1469150/ns/pid -- /usr/local/bin/tproxy -i -vv
root 1476117 0.0 0.0 549552 6348 ? Sl 17:13 0:00 /usr/local/bin/tproxy -i -vv
root 1476833 0.0 0.0 3084 884 pts/0 S+ 17:15 0:00 grep tproxy
The logs:
Mar 07 17:13:04.593 INFO Proxy executor: Starting proxy.
Mar 07 17:13:04.593 INFO Uds listener listening on: "/tmp/e1270471-edb5-497a-81cb-024e06c6e23e.sock"
2022-03-07T17:13:04.593Z INFO chaos-daemon.daemon-server chaosdaemon/httpchaos_server.go:152 http chaos applied {"namespacedName": "default/nginx-848f97569b-s5dhv"}
Mar 07 17:13:04.594 INFO Proxy executor: Proxy is running.
Mar 07 17:13:04.622 INFO Proxy get uds path: "/tmp/e1270471-edb5-497a-81cb-024e06c6e23e.sock"
Mar 07 17:13:04.623 DEBUG rs_tproxy_proxy::uds_client: try connect path : "/tmp/e1270471-edb5-497a-81cb-024e06c6e23e.sock"
Mar 07 17:13:04.623 INFO Uds server: Config successfully transferred.
Mar 07 17:13:04.623 DEBUG rs_tproxy_proxy::uds_client: Read data successfully.
Mar 07 17:13:04.623 DEBUG rs_tproxy_proxy::uds_client: Deserialize data successfully.
Mar 07 17:13:04.624 INFO Proxy: Starting
Mar 07 17:13:04.624 INFO Proxy: Listening
2022-03-07T17:13:05.713Z INFO chaos-daemon.daemon-server chaosdaemon/container.go:52 container GetPi{"request": "action:{action:GETPID} container_id:\"docker://1379852f69396474ad1b9360efdb95992c7558e8796969e378fc8e3bd2018d30\""}
2022-03-07T17:14:12.471Z INFO chaos-daemon.daemon-server chaosdaemon/container.go:52 container GetPi{"request": "action:{action:GETPID} container_id:\"docker://1379852f69396474ad1b9360efdb95992c7558e8796969e378fc8e3bd2018d30\""}
2022-03-07T17:14:12.546Z INFO chaos-daemon.daemon-server chaosdaemon/container.go:52 container GetPi{"request": "action:{action:GETPID} container_id:\"docker://1379852f69396474ad1b9360efdb95992c7558e8796969e378fc8e3bd2018d30\""}
2022-03-07T17:14:12.611Z INFO chaos-daemon.daemon-server chaosdaemon/container.go:52 container GetPi{"request": "action:{action:GETPID} container_id:\"docker://1379852f69396474ad1b9360efdb95992c7558e8796969e378fc8e3bd2018d30\""}
Mar 07 17:14:12.678 INFO Proxy executor: killing sub process
Mar 07 17:14:12.719 DEBUG rs_tproxy::proxy::net::bridge: stderr : RTNETLINK answers: File exists
Mar 07 17:14:12.722 DEBUG rs_tproxy::proxy::net::bridge: stderr : RTNETLINK answers: File exists
Mar 07 17:14:12.724 DEBUG rs_tproxy::proxy::net::bridge: stderr : RTNETLINK answers: File exists
Mar 07 17:14:12.726 DEBUG rs_tproxy::proxy::net::bridge: stderr : RTNETLINK answers: File exists
Mar 07 17:14:12.728 DEBUG rs_tproxy::proxy::net::bridge: stderr : RTNETLINK answers: File exists
Mar 07 17:14:12.730 DEBUG rs_tproxy::proxy::net::bridge: stderr : RTNETLINK answers: File exists
Mar 07 17:14:12.732 DEBUG rs_tproxy::proxy::net::bridge: stderr : RTNETLINK answers: File exists
This issue doesn't cause any functional effects on Chaos Mesh as the proxy of chaos-tproxy can be killed gracefully, but it causes the graceful shutdown of chaos-daemon very slow.