Set up clustering
dnsbty opened this issue · comments
Elixir's Registry only works on the local process. Swap it out with Erlang's :global to set things up for clustering and make scaling possible.
:global docs: http://erlang.org/doc/man/global.html
Relevant blog post: https://scottyscripts.com/2019/11/25/erlang-in-elixir-erlangs-global-module.html
Relevant Elixir Forum thread: https://elixirforum.com/t/genserver-and-global-registration/13344
Perhaps longer term, syn should be considered instead based on these benchmarks but it seems like until we're seeing a ton of traffic, :global ought to work just fine
Another blog post about global: https://moosecode.nl/blog/highlander_there_can_be_only_one
After more research, it appears that swarm will actually be a better option for registering global processes because it will also automatically balance them across nodes for us.
I'm not positive, but hopeful that it will allow us to also preserve state across nodes whenever we're deploying, but I'm not positive about that.
After even more research, it appears that horde is what we're really looking for. It will allow us to do everything swarm will, but should also allow us to move state to other nodes whenever we do a graceful shutdown.
https://github.com/derekkraan/horde
https://hexdocs.pm/horde/getting_started.html
https://metasyntactic.info/distributing-phoenix-part-2/
Then we can use libcluster for actually setting up the cluster within gigalixir
It's looking like Horde is definitely the way to go. Here are some more resources:
Amazing talk about how he did basically exactly what we want to do: https://daniel-azuma.com/articles/talks/elixirconf-2018
https://engineering.dollarshaveclub.com/elixir-otp-applications-on-kubernetes-9944636b8609
https://medium.com/@ellispritchard/graceful-shutdown-on-kubernetes-with-signals-erlang-otp-20-a22325e8ae98
As of 2e24723 Horde is in place so connected nodes will be able to take advantage of clustering 🎉
Before I close this issue, I would like to get libcluster set up for automatic clustering in gigalixir so that we can easily flip that switch.
I would also like to use Horde to transfer process state between nodes whenever a sigterm is received.
As of 267e4eb libcluster is in place to automatically connect nodes.
The only thing left now is to use DeltaCRDT to hand off state between nodes whenever a SIGTERM is received. The article here seems to be the best reference on how to do this: https://engineering.dollarshaveclub.com/elixir-otp-applications-on-kubernetes-9944636b8609#de27
With bcab6c8 we're now transferring state between processes when running everything locally.
Unfortunately it appears that Gigalixir isn't allowing that to work the way I would expect, so I will create a new ticket to figure out what's going on there and take care of it.
The basis for this is complete. The new ticket is #67