properly handle signals (SIGINT, SIGTERM) to terminate the server correctly
andy108369 opened this issue · comments
A shell script is running the binary, when the container receives SIGTERM (say on Pod termination or docker stop / kill with the default SIGTERM, or Ctrl+C - SIGINT) the shell script does not forward these signals to the binary.
You can quickly test and see SIGTERM is not working, just issue docker [compose] stop -t600 node it won't do anything, until it times out and issues the SIGKILL which is not stopping the binary gracefully.
Cosmos SDK understands SIGTERM & SIGINT for graceful server termination.
Signal handling can be done directly in the shell script, in cosmos-omnibus that would be both run.sh and snapshot.sh.
We can reuse the signal handling from one of my projects I've been working on - self-update.
Workaround
Get into the container and then kill <pid-of-cosmos-binary> (issues SIGTERM by default).
could probably leverage the wait -n method (it waits for any process ./process1 &, ./process2 &, ... to exit in the main script) https://docs.docker.com/config/containers/multi-service_container/
if it helps in any way after modifying the run.sh & snapshot.sh scripts to handle the signals.