/etc/hosts
management with Serf
Automatic Problem
In the cloud world, many hosts appear and vanish. Since we don't want to bother to manage some internal DNS service while keeping its availability, we have been updating /etc/hosts
file periodically with cron and AWS API.
There are, however, several problems in that way of updating /etc/hosts
:
- It's far from real time
- There are many other components that need to be updated in a likely way; for example, nagios, munin, etc.
It's laborious that we have to write scripts for each purposes and edit crontab.
Solution
Serf can solve the problem. It provides us decentralized hosts discovery solution. Once we launch serf agents in each hosts, the cluster itself works like it is managed completely. In addition, it's almost real time.
Example
For Mac OS X uses, please check this document to circumvent a problem in OS X at first.
Launch the First Node
Launch a serf agent named node1
:
$ serf agent -node=node1 -bind=127.0.0.10 -rpc-addr=127.0.0.1:7374 -event-handler=event_handler.pl
Then you'll see etc/hosts
to be like below:
$ cat etc/hosts
127.0.0.10 node1
Launch the Second Node
Launch another node named node2
:
$ serf agent -node=node2 -bind=127.0.0.11 -rpc-addr=127.0.0.1:7375
Then add node2
into the cluster that currently consists of only node1
:
$ serf join -rpc-addr=127.0.0.1:7374 127.1.0.11
As a result,
node2
is now also a member of the clustermember-join
event is emittednode2
is added intoetc/hosts
$ cat etc/hosts
127.0.0.10 node1
127.0.0.11 node2
Remove the Second Node from the Cluster
Push Ctrl-C
to stop node2
. Then member-leave
event is propagated to node1
and the handler script is fired.
As a result,
node2
is now not a member of the clustermember-leave
event is emittednode2
is removed frometc/hosts
$ cat etc/hosts
127.0.0.10 node1