Giters

torodb / stampede

The ToroDB solution to provide better analytics on top of MongoDB and make it easier to migrate from MongoDB to SQL

Home Page:https://www.torodb.com/stampede/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

RecoveryService fatal error

p1nox opened this issue · comments

commented

I have three containers, one for postgres and two for mongo, mongo1 and mongo2 replica sets:

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                     PORTS                      NAMES
00aa057a85c5        mongo               "/entrypoint.sh mongo"   13 minutes ago      Up 13 minutes              0.0.0.0:27018->27017/tcp   mongo2
735ed126f61e        mongo               "/entrypoint.sh mongo"   13 minutes ago      Up 13 minutes              0.0.0.0:27017->27017/tcp   mongo1
e4ec4ea1dd1d        postgres:9.5        "/docker-entrypoint.s"   50 minutes ago      Up 50 minutes              0.0.0.0:5432->5432/tcp     postgres_test

Also, I already have postgres database configured with the requirements: torodb and torod databases and torodb user.

I start mongo containers like:

# create a custom network
docker network create my-mongo-cluster

# start first replica using this network
docker run -d \
-p 27017:27017 \
--name mongo1 \
--net my-mongo-cluster \
mongo mongod --replSet rs1

# start second replica using same network
docker run -d \
-p 27018:27017 \
--name mongo2 \
--net my-mongo-cluster \
mongo mongod --replSet rs1

I start stampede container like:

docker run -ti -v ~/.toropass:/root/.toropass --net my-mongo-cluster torodb/stampede

So when stampede container is starting, everything seams fine, all service connected, but then I receive an error message, and container is shutdown:

2016-12-22T04:01:25.114 INFO 'StampedeService STARTING' c.t.s.StampedeService Starting up ToroDB Stampede
2016-12-22T04:01:25.127 INFO 'StampedeService STARTING' c.t.b.p.PostgreSqlDbBackend Configured PostgreSQL backend at 172.22.0.1:5432
2016-12-22T04:01:25.701 INFO 'PostgreSqlDbBackend STARTING' c.t.b.AbstractDbBackendService Created pool session with size 28 and level TRANSACTION_REPEATABLE_READ
2016-12-22T04:01:25.743 INFO 'PostgreSqlDbBackend STARTING' c.t.b.AbstractDbBackendService Created pool system with size 1 and level TRANSACTION_REPEATABLE_READ
2016-12-22T04:01:25.756 INFO 'PostgreSqlDbBackend STARTING' c.t.b.AbstractDbBackendService Created pool cursors with size 1 and level TRANSACTION_REPEATABLE_READ
2016-12-22T04:01:26.329 INFO 'BackendBundleImpl STARTING' c.t.b.m.AbstractSchemaUpdater Schema 'torodb' found. Checking it...
2016-12-22T04:01:26.400 INFO 'BackendBundleImpl STARTING' c.t.b.m.AbstractSchemaUpdater Schema 'torodb' checked
2016-12-22T04:01:26.563 INFO 'StampedeService STARTING' c.t.s.StampedeService Database is not consistent. Cleaning it up
2016-12-22T04:01:26.646 INFO 'StampedeService STARTING' c.t.s.StampedeService Replicating from seeds: 172.22.0.1:27017
2016-12-22T04:01:27.039 INFO 'MongodbReplBundle STARTING' c.t.m.r.MongodbReplBundle Starting replication service
2016-12-22T04:01:27.271 INFO 'topology-executor-0' c.t.m.r.t.TopologyCoordinator Waiting for 4  pings from other members before syncing
2016-12-22T04:01:27.295 INFO 'topology-executor-0' c.t.m.c.p.MemberHeartbeatData Member mongo2:27017 is now in state RS_SECONDARY
2016-12-22T04:01:27.297 INFO 'topology-executor-0' c.t.m.c.p.MemberHeartbeatData Member mongo1:27017 is now in state RS_PRIMARY
2016-12-22T04:01:28.273 INFO 'topology-executor-0' c.t.m.r.t.TopologyCoordinator Waiting for 2  pings from other members before syncing
2016-12-22T04:01:29.274 INFO 'topology-executor-0' c.t.m.r.t.TopologyCoordinator Waiting for 2  pings from other members before syncing
2016-12-22T04:01:30.282 INFO 'topology-executor-0' c.t.m.r.t.TopologyCoordinator syncing from: mongo2:27017
2016-12-22T04:01:30.283 INFO 'TopologyService STARTING' c.t.m.r.t.TopologyService Topology service started
2016-12-22T04:01:30.331 WARN 'ReplCoordinator STARTING' c.t.m.r.ReplCoordinator loadStoredConfig() is not implemented yet
2016-12-22T04:01:30.331 INFO 'ReplCoordinator STARTING' c.t.m.r.ReplCoordinator Database is not consistent.
2016-12-22T04:01:30.334 INFO 'MongodbReplBundle STARTING' c.t.m.r.MongodbReplBundle Replication service started
2016-12-22T04:01:30.334 INFO 'StampedeService STARTING' c.t.s.StampedeService ToroDB Stampede is now running
2016-12-22T04:01:30.336 INFO 'repl-coord-starting-recovery' c.t.m.r.ReplCoordinatorStateMachine Starting RECOVERY mode
2016-12-22T04:01:30.364 INFO 'RecoveryService' c.t.m.r.RecoveryService Starting RECOVERY service
2016-12-22T04:01:30.364 INFO 'RecoveryService' c.t.m.r.RecoveryService Starting initial sync
2016-12-22T04:01:30.400 INFO 'RecoveryService' c.t.s.DefaultConsistencyHandler Consistent state has been set to 'false'
2016-12-22T04:01:30.405 INFO 'RecoveryService' c.t.m.r.RecoveryService Using node mongo2:27017 to replicate from
2016-12-22T04:01:30.434 INFO 'RecoveryService' c.t.m.r.RecoveryService Remote database cloning started
2016-12-22T04:01:30.591 INFO 'RecoveryService' c.t.b.SharedWriteBackendTransactionImpl Created internal index did_pkey for table oplog_replication
2016-12-22T04:01:30.635 INFO 'RecoveryService' c.t.b.SharedWriteBackendTransactionImpl Created internal index rid_pkey for table oplog_replication_lastappliedoplogentry
2016-12-22T04:01:30.668 INFO 'RecoveryService' c.t.b.SharedWriteBackendTransactionImpl Created internal index did_idx for table oplog_replication_lastappliedoplogentry
2016-12-22T04:01:30.698 INFO 'RecoveryService' c.t.b.SharedWriteBackendTransactionImpl Created internal index did_seq_idx for table oplog_replication_lastappliedoplogentry
2016-12-22T04:01:30.730 INFO 'RecoveryService' c.t.m.r.RecoveryService Local databases dropping started
2016-12-22T04:01:30.756 INFO 'RecoveryService' c.t.m.r.RecoveryService Local databases dropping finished
2016-12-22T04:01:30.756 INFO 'RecoveryService' c.t.m.r.RecoveryService Remote database cloning started
2016-12-22T04:01:30.851 ERROR 'RecoveryService' c.t.m.r.ReplCoordinator Fatal error while starting recovery mode: Not supported yet
2016-12-22T04:01:30.854 ERROR 'RecoveryService' c.t.m.r.MongodbReplBundle Catched an error on the replication layer. Escalating it
2016-12-22T04:01:30.855 ERROR 'RecoveryService' c.t.s.StampedeService Error reported by replication supervisor. Stopping ToroDB Stampede
2016-12-22T04:01:30.858 INFO 'StampedeService STOPPING' c.t.s.StampedeService Shutting down ToroDB Stampede
2016-12-22T04:01:30.858 INFO 'RecoveryService' c.t.m.r.RecoveryService Recived a request to stop the recovering service
2016-12-22T04:01:30.906 INFO 'MongodbReplBundle STOPPING' c.t.m.r.MongodbReplBundle Shutting down replication service
2016-12-22T04:01:30.924 INFO 'TopologyService STOPPING' c.t.m.r.t.TopologyService Topology service shutted down
2016-12-22T04:01:30.937 INFO 'MongodbReplBundle STOPPING' c.t.m.r.MongodbReplBundle Replication service shutted down
2016-12-22T04:01:31.357 INFO 'StampedeService STOPPING' c.t.s.StampedeService ToroDB Stampede has been shutted down

Where the specific errors seams related to RecoveryService:

2016-12-22T04:01:30.851 ERROR 'RecoveryService' c.t.m.r.ReplCoordinator Fatal error while starting recovery mode: Not supported yet
2016-12-22T04:01:30.854 ERROR 'RecoveryService' c.t.m.r.MongodbReplBundle Catched an error on the replication layer. Escalating it
2016-12-22T04:01:30.855 ERROR 'RecoveryService' c.t.s.StampedeService Error reported by replication supervisor. Stopping ToroDB Stampede
commented

Hi @p1nox,

You should specify mongo:3.2 as the mongo image since right now we only support that version. Hopefully, support for 3.4 will be out anytime soon.

commented

Great! My bad, thanks @teoincontatto, that was the problem indeed 💯