feature: custom gossip for member list
zenyui opened this issue · comments
ideally, we could implement a simple ping gossip method for joining and maintaining the peer member list. currently, we use hashicorp/memberlist, but that requires another port to be open, and it's noisy.
forming the cluster
we could implement our own member handler that receives peers from the DiscoveryService implementation and calls a Ping method on that peer. if the discovered node already has peers, it could share the whole list with the asking node to bootstrap the cluster
maintaining a cluster
For established peers, a background gossip loop could run where each node continuously health checks other nodes, and if multiple nodes are unable to ping the same peer, the leader can remove that peer from the raft cluster. perhaps the leader could coordinate asking all nodes to ping a specific node and tally the results centrally, or each raft node could randomly ping its peers and notify the leader of any failures.