Dalee / mongodb-ha-playground

Minimal MongoDB 3.2 High Availability setup with failover and correct load balancing.

Minimal MongoDB 3.2 High Availability setup

Vagrant based playground for setting up two MongoDB instances with replication, failover and correct load balancing.

setting up

$ vagrant up
$ vagrant ssh
$ /vagrant/setup.sh

Initial configuration:

• domains db1.example.com, db2.example.com and arbiter1.example.com are aliases for 127.0.0.1
• mongodb on db1.example.com:27017 is primary
• mongodb on db2.example.com:27018 is secondary
• mongodb on arbiter1.example.com:47017 is arbiter
• haproxy on port 27016 is mongodb balancer
• replSetName is rs0 (it's just name for mongo-ha replication instances)
• electionTimeoutMillis is set to 2s

Each run of setup.sh script, reverts state to initial configuration.

checking setup

check current primary node:

$ mongo --quiet --port 27016 --eval "db.isMaster()"
{
	"hosts" : [
		"db1.example.com:27017",
		"db2.example.com:27018"
	],
	"arbiters" : [
		"arbiter1.example.com:47017"
	],
	"setName" : "rs0",
	"setVersion" : 1,
	"ismaster" : true,
	"secondary" : false,
	"primary" : "db1.example.com:27017",
	"me" : "db1.example.com:27017",
	"electionId" : ObjectId("7fffffff0000000000000001"),
	"maxBsonObjectSize" : 16777216,
	"maxMessageSizeBytes" : 48000000,
	"maxWriteBatchSize" : 1000,
	"localTime" : ISODate("2017-09-06T17:22:16.887Z"),
	"maxWireVersion" : 4,
	"minWireVersion" : 0,
	"ok" : 1
}

create sample database and collection via haproxy:

$ mongo --quiet --port 27016 --eval 'db.hello.insert({"name":"world"});' test-db
WriteResult({ "nInserted" : 1 })

ensure test collection is exists and accessible via haproxy:

$ mongo --quiet --port 27016 --eval "db.hello.find()" test-db
{ "_id" : ObjectId("59b04b541e80cc01f7d66afd"), "name" : "world" }

testing failover

kill current primary node:

$ kill -9 $(mongo --quiet --port 27017 --eval "JSON.stringify(db.serverStatus())" | jq '.pid.floatApprox')

wait for few seconds (should happen within electionTimeoutMillis)...

ensure current primary node is on port 27018 and accessible via haproxy:

$ mongo --quiet --port 27016 --eval "db.isMaster()"
{
	"hosts" : [
		"db1.example.com:27017",
		"db2.example.com:27018"
	],
	"arbiters" : [
		"arbiter1.example.com:47017"
	],
	"setName" : "rs0",
	"setVersion" : 1,
	"ismaster" : true,
	"secondary" : false,
	"primary" : "db2.example.com:27018",
	"me" : "db2.example.com:27018",
	"electionId" : ObjectId("7fffffff0000000000000002"),
	"maxBsonObjectSize" : 16777216,
	"maxMessageSizeBytes" : 48000000,
	"maxWriteBatchSize" : 1000,
	"localTime" : ISODate("2017-09-06T17:23:38.045Z"),
	"maxWireVersion" : 4,
	"minWireVersion" : 0,
	"ok" : 1
}

ensure test collection and database exists and accessible via haproxy:

$ mongo --quiet --port 27016 --eval "db.hello.find()" test-db
{ "_id" : ObjectId("59b0305101c76adf9a0cb8ee"), "name" : "world" }

return failed mongodb node back

start killed node:

$ sudo systemctl start mongodb-27017

ensure mongodb node on port 27017 is secondary:

$ mongo --quiet --port 27017 --eval "db.isMaster()"
{
	"hosts" : [
		"db1.example.com:27017",
		"db2.example.com:27018"
	],
	"arbiters" : [
		"arbiter1.example.com:47017"
	],
	"setName" : "rs0",
	"setVersion" : 1,
	"ismaster" : false,
	"secondary" : true,
	"primary" : "db2.example.com:27018",
	"me" : "db1.example.com:27017",
	"maxBsonObjectSize" : 16777216,
	"maxMessageSizeBytes" : 48000000,
	"maxWriteBatchSize" : 1000,
	"localTime" : ISODate("2017-09-06T19:29:12.940Z"),
	"maxWireVersion" : 4,
	"minWireVersion" : 0,
	"ok" : 1
}