chaos-tproxy-controller cannot graceful shut down

Question

chaos-tproxy-controller cannot graceful shut down

Hexilee opened this issue 2 years ago · comments

After chaos-mesh/chaos-mesh#2668, we have the graceful shutdown in Chaos Mesh. However, I found the controller of chaos-tproxy cannot be killed by SIGINT or SIGTERM.

root@chaos-daemon-nc2p5:/# ps aux | grep tproxy
root     1476116  0.0  0.0   3388  2496 ?        S    17:13   0:00 /usr/local/bin/nsexec -l -n /proc/1469150/ns/net -p /proc/1469150/ns/pid -- /usr/local/bin/tproxy -i -vv
root     1476117  0.0  0.0 549552  6348 ?        Sl   17:13   0:00 /usr/local/bin/tproxy -i -vv
root     1476779  0.0  0.0   3084   888 pts/0    R+   17:14   0:00 grep tproxy
root@chaos-daemon-nc2p5:/# kill 1476117
root@chaos-daemon-nc2p5:/# ps aux | grep tproxy
root     1476116  0.0  0.0   3388  2496 ?        S    17:13   0:00 /usr/local/bin/nsexec -l -n /proc/1469150/ns/net -p /proc/1469150/ns/pid -- /usr/local/bin/tproxy -i -vv
root     1476117  0.0  0.0 549552  6348 ?        Sl   17:13   0:00 /usr/local/bin/tproxy -i -vv
root     1476805  0.0  0.0   3084   892 pts/0    S+   17:14   0:00 grep tproxy
root@chaos-daemon-nc2p5:/# ps aux | grep tproxy
root     1476116  0.0  0.0   3388  2496 ?        S    17:13   0:00 /usr/local/bin/nsexec -l -n /proc/1469150/ns/net -p /proc/1469150/ns/pid -- /usr/local/bin/tproxy -i -vv
root     1476117  0.0  0.0 549552  6348 ?        Sl   17:13   0:00 /usr/local/bin/tproxy -i -vv
root     1476807  0.0  0.0   3084   888 pts/0    S+   17:14   0:00 grep tproxy
root@chaos-daemon-nc2p5:/# kill -15 1476117
root@chaos-daemon-nc2p5:/# ps aux | grep tproxy
root     1476116  0.0  0.0   3388  2496 ?        S    17:13   0:00 /usr/local/bin/nsexec -l -n /proc/1469150/ns/net -p /proc/1469150/ns/pid -- /usr/local/bin/tproxy -i -vv
root     1476117  0.0  0.0 549552  6348 ?        Sl   17:13   0:00 /usr/local/bin/tproxy -i -vv
root     1476833  0.0  0.0   3084   884 pts/0    S+   17:15   0:00 grep tproxy

The logs:

Mar 07 17:13:04.593  INFO Proxy executor: Starting proxy.
Mar 07 17:13:04.593  INFO Uds listener listening on: "/tmp/e1270471-edb5-497a-81cb-024e06c6e23e.sock"
2022-03-07T17:13:04.593Z        INFO    chaos-daemon.daemon-server      chaosdaemon/httpchaos_server.go:152     http chaos applied      {"namespacedName": "default/nginx-848f97569b-s5dhv"}
Mar 07 17:13:04.594  INFO Proxy executor: Proxy is running.
Mar 07 17:13:04.622  INFO Proxy get uds path: "/tmp/e1270471-edb5-497a-81cb-024e06c6e23e.sock"
Mar 07 17:13:04.623 DEBUG rs_tproxy_proxy::uds_client: try connect path : "/tmp/e1270471-edb5-497a-81cb-024e06c6e23e.sock"
Mar 07 17:13:04.623  INFO Uds server: Config successfully transferred.
Mar 07 17:13:04.623 DEBUG rs_tproxy_proxy::uds_client: Read data successfully.
Mar 07 17:13:04.623 DEBUG rs_tproxy_proxy::uds_client: Deserialize data successfully.
Mar 07 17:13:04.624  INFO Proxy: Starting
Mar 07 17:13:04.624  INFO Proxy: Listening
2022-03-07T17:13:05.713Z        INFO    chaos-daemon.daemon-server      chaosdaemon/container.go:52     container GetPi{"request": "action:{action:GETPID}  container_id:\"docker://1379852f69396474ad1b9360efdb95992c7558e8796969e378fc8e3bd2018d30\""}
2022-03-07T17:14:12.471Z        INFO    chaos-daemon.daemon-server      chaosdaemon/container.go:52     container GetPi{"request": "action:{action:GETPID}  container_id:\"docker://1379852f69396474ad1b9360efdb95992c7558e8796969e378fc8e3bd2018d30\""}
2022-03-07T17:14:12.546Z        INFO    chaos-daemon.daemon-server      chaosdaemon/container.go:52     container GetPi{"request": "action:{action:GETPID}  container_id:\"docker://1379852f69396474ad1b9360efdb95992c7558e8796969e378fc8e3bd2018d30\""}
2022-03-07T17:14:12.611Z        INFO    chaos-daemon.daemon-server      chaosdaemon/container.go:52     container GetPi{"request": "action:{action:GETPID}  container_id:\"docker://1379852f69396474ad1b9360efdb95992c7558e8796969e378fc8e3bd2018d30\""}
Mar 07 17:14:12.678  INFO Proxy executor: killing sub process
Mar 07 17:14:12.719 DEBUG rs_tproxy::proxy::net::bridge: stderr : RTNETLINK answers: File exists

Mar 07 17:14:12.722 DEBUG rs_tproxy::proxy::net::bridge: stderr : RTNETLINK answers: File exists

Mar 07 17:14:12.724 DEBUG rs_tproxy::proxy::net::bridge: stderr : RTNETLINK answers: File exists

Mar 07 17:14:12.726 DEBUG rs_tproxy::proxy::net::bridge: stderr : RTNETLINK answers: File exists

Mar 07 17:14:12.728 DEBUG rs_tproxy::proxy::net::bridge: stderr : RTNETLINK answers: File exists

Mar 07 17:14:12.730 DEBUG rs_tproxy::proxy::net::bridge: stderr : RTNETLINK answers: File exists

Mar 07 17:14:12.732 DEBUG rs_tproxy::proxy::net::bridge: stderr : RTNETLINK answers: File exists

This issue doesn't cause any functional effects on Chaos Mesh as the proxy of chaos-tproxy can be killed gracefully, but it causes the graceful shutdown of chaos-daemon very slow.