Minimalistic redis sentinel process only to do slave promoton
If you are running redis-servers on infrastructues that are managed by cluster managers like kubernetes or apache mesos, these cluster managers automatically take care of redis-server's life cycle, we dont need sentinel for that. The only thing that would be missing is slave promotion incase the master crashed. We want an utility only for that functionality.
This project follows the same approach that is followed by the redis-sentinel in function compareSlavesForPromotion()
int compareSlavesForPromotion(const void *a, const void *b) {
/*Step 1: Lowest Priority. */
/*Step 2: Maximum replication offset. */
/* Step 3: Lexographic sort based on runid. */
}
except Step 3, where instead of Lexographic redis-sentinel-micro selects the slave which completed replication most recently, based on master_last_io_seconds_ago attribute.
Assume a Redis Instance with 1 Master and 5 Slaves, like below
role:master
connected_slaves:5
slave0:ip=172.31.10.90,port=6385,state=online,offset=197,lag=1
slave1:ip=172.31.10.90,port=6386,state=online,offset=197,lag=1
slave2:ip=172.31.10.90,port=6384,state=online,offset=197,lag=1
slave3:ip=172.31.10.90,port=6383,state=online,offset=197,lag=1
slave4:ip=172.31.10.90,port=6382,state=online,offset=197,lag=1
master_repl_offset:197
The master crashes mid-way leaving all the 5 slaves in an inconsistent stage like below
slave_repl_offset:98187702
slave_repl_offset:98187702
slave_repl_offset:98150330
slave_repl_offset:98134252
slave_repl_offset:98134252
All the above slaves have equal priority, but we have 3 replication values 98134252, 98150330 and 98187702. Ideally the slave with highest replication value should be promoted.
When you run redis-sentinel-micro like this it automatically detects the next master and re-configures all the slaves to point to this new master
$./redis-sentinel-micro -logtostderr 0.0.0.0:6382 0.0.0.0:6383 0.0.0.0:6384 0.0.0.0:6385 0.0.0.0:6386
I1213 19:19:58.530901 16261 redis_sentinel_micro.go:290] Supplied Arguments are [0.0.0.0:6382 0.0.0.0:6383 0.0.0.0:6384 0.0.0.0:6385 0.0.0.0:6386]
I1213 19:19:58.531001 16261 redis_sentinel_micro.go:161] Processing 0.0.0.0:6382
I1213 19:19:58.531042 16261 redis_sentinel_micro.go:161] Processing 0.0.0.0:6383
I1213 19:19:58.531073 16261 redis_sentinel_micro.go:161] Processing 0.0.0.0:6384
I1213 19:19:58.531102 16261 redis_sentinel_micro.go:161] Processing 0.0.0.0:6385
I1213 19:19:58.531135 16261 redis_sentinel_micro.go:161] Processing 0.0.0.0:6386
R=&{ slave -1 610 98150330 172.31.10.90 6381 100 false <nil>}
R=&{ slave -1 610 98134252 172.31.10.90 6381 100 false <nil>}
R=&{ slave -1 610 98187702 172.31.10.90 6381 100 false <nil>}
R=&{ slave -1 610 98134252 172.31.10.90 6381 100 false <nil>}
R=&{ slave -1 610 98187702 172.31.10.90 6381 100 false <nil>}
I1213 19:19:58.532407 16261 redis_sentinel_micro.go:275] PrintServers()
<<SNIPPED prints an elaborate json array>>
I1213 19:19:58.533968 16261 redis_sentinel_micro.go:303] New Master={0.0.0.0:6382 slave -1 610 98187702 172.31.10.90 6381 100 false 0xc82007e300}
I1213 19:19:58.534858 16261 redis_sentinel_micro.go:312] New Master is 0.0.0.0:6382, All the slaves are re-configured to replicate from this
I1213 19:19:58.534986 16261 redis_sentinel_micro.go:275] PrintServers()
<<SNIPPED prints an elaborate json array>>
I1213 19:19:58.536414 16261 redis_sentinel_micro.go:316] Redis-Sentinal-micro Finished
The slave 6382 becomes the new master
./redis-cli -h 0.0.0.0 -p 6382 info replication
# Replication
role:master
connected_slaves:4
slave0:ip=127.0.0.1,port=6385,state=online,offset=631,lag=1
slave1:ip=127.0.0.1,port=6384,state=online,offset=631,lag=0
slave2:ip=127.0.0.1,port=6383,state=online,offset=631,lag=1
slave3:ip=127.0.0.1,port=6386,state=online,offset=631,lag=0
master_repl_offset:631
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:630
If you run it on a stable (a properly configured) cluster, it does nothing
./redis-sentinel-micro -logtostderr 0.0.0.0:6382 0.0.0.0:6383 0.0.0.0:6384 0.0.0.0:6385 0.0.0.0:6386
I1213 19:29:12.034424 16369 redis_sentinel_micro.go:290] Supplied Arguments are [0.0.0.0:6382 0.0.0.0:6383 0.0.0.0:6384 0.0.0.0:6385 0.0.0.0:6386]
I1213 19:29:12.034517 16369 redis_sentinel_micro.go:161] Processing 0.0.0.0:6382
I1213 19:29:12.034568 16369 redis_sentinel_micro.go:161] Processing 0.0.0.0:6383
I1213 19:29:12.034602 16369 redis_sentinel_micro.go:161] Processing 0.0.0.0:6384
I1213 19:29:12.034636 16369 redis_sentinel_micro.go:161] Processing 0.0.0.0:6385
I1213 19:29:12.034681 16369 redis_sentinel_micro.go:161] Processing 0.0.0.0:6386
R=&{ master 0 0 771 0 false <nil>}
R=&{ slave 7 0 771 0.0.0.0 6382 100 false <nil>}
R=&{ slave 7 0 771 0.0.0.0 6382 100 false <nil>}
R=&{ slave 7 0 771 0.0.0.0 6382 100 false <nil>}
R=&{ slave 7 0 771 0.0.0.0 6382 100 false <nil>}
I1213 19:29:12.036024 16369 redis_sentinel_micro.go:275] PrintServers()
<<SNIPPED prints an elaborate json array>>
I1213 19:29:12.037455 16369 redis_sentinel_micro.go:205] RSMaster_EP=0.0.0.0:6382 available MasterEP=0.0.0.0:6382
I1213 19:29:12.037498 16369 redis_sentinel_micro.go:205] RSMaster_EP=0.0.0.0:6382 available MasterEP=0.0.0.0:6382
I1213 19:29:12.037534 16369 redis_sentinel_micro.go:205] RSMaster_EP=0.0.0.0:6382 available MasterEP=0.0.0.0:6382
I1213 19:29:12.037558 16369 redis_sentinel_micro.go:205] RSMaster_EP=0.0.0.0:6382 available MasterEP=0.0.0.0:6382
W1213 19:29:12.037581 16369 redis_sentinel_micro.go:220] The redis master is already configured, dont do anything SyncBytes=771 availableMasterHits=4 len(Slaves)=4
E1213 19:29:12.037606 16369 redis_sentinel_micro.go:300] Redis Instance does'nt need a Slave Promotion
All building can happen in docker to keep from requiring build dependencies on local machine.
To build a new image for the sentinel binary, run the following command:
docker build -t dhilipkumars/redis-sentinel-k8s:0.2.0 ./
To build a new image for the make_slave binary, run the following command:
docker build -t dhilipkumars/mk-redis-slave:0.2.0 -f mk_redis_slave/Dockerfile ./mk_redis_slave