Fatal error while starting recovery mode: Error while cloning indexes: null

[sk]

Hello,

I've followed the installation and setup and things appear to be running well. However, during the migration, it seems to fail with the following stack traced (truncated for brevity):

2017-09-06T03:39:22.362 INFO REPL - Cloning collection indexes mulch.users into mulch.users 2017-09-06T03:39:22.363 INFO REPL - Index users.mulch._id_ will be cloned 2017-09-06T03:39:22.363 INFO REPL - Index users.mulch.firstname_1_lastname_1_email_1_uid_1 will be cloned 2017-09-06T03:39:22.363 INFO REPL - Index users.mulch.uid_1 will be cloned 2017-09-06T03:39:22.363 INFO REPL - Index users.mulch.email_1 will be cloned 2017-09-06T03:39:22.365 ERROR REPL - Fatal error while starting recovery mode: Error while cloning indexes: null 2017-09-06T03:39:22.368 ERROR REPL - Catched an error on the replication layer. Escalating it 2017-09-06T03:39:22.368 ERROR LIFECYCLE - Error reported by replication supervisor. Stopping ToroDB Stampede 2017-09-06T03:39:22.370 INFO LIFECYCLE - Shutting down ToroDB Stampede 2017-09-06T03:39:22.370 INFO REPL - Recived a request to stop the recovering service 2017-09-06T03:39:22.373 INFO REPL - Shutting down replication service 2017-09-06T03:39:22.420 INFO REPL - Topology service shutted down 2017-09-06T03:39:22.423 INFO REPL - Replication service shutted down 2017-09-06T03:39:23.392 INFO LIFECYCLE - ToroDB Stampede has been shutted down

It actually seems to work and by that I mean in the PG schema mulch are ALL the tables (way more than there should be), but the core tables are migrated and the data is there.

Can you give me any information on the error. I googled and didn't see anything of value.
Thanks!

[Matteo Melli]

Hello @mr-sk,

Actual beta3 release suffer a bug that breaks the recovery when an unsupported index is found (in this case the compound index: users.mulch.firstname_1_lastname_1_email_1_uid_1). A workaround is to define a filter that exclude all the indexes (see also https://www.torodb.com/stampede/docs/1.0.0-beta3/configuration/filtered-replication/#exclude-ignore-a-database-collections-or-indexes):

replication:
  excludes:
    "*":
      "*":
        - name: *

Another solution is to use the development version where that bug is fixed. Our documentation include instruction to install the nigthly build for various platform (see here https://www.torodb.com/stampede/docs/1.0.0-beta3/).

[sk]

Hey @teoincontatto,

Thanks for the response. I'm attempting to build the nightly version but I'm experiencing issues. I opened another issue.

Thanks and I'll report back once I get that fixed.

[Jess Kane]

Hey @teoincontatto,

I'm a coworker of @mr-sk, picking up where he left off. For the beta3 release, excluding indexes via a config file did solve the initial problem. However, we're now experiencing a new error relating to duplicate primary keys:

com.torodb.core.transaction.RollbackException: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "field_pkey"
  Detail: Key (database, collection, table_ref, name, type)=(mulch, users, {}, _id, MONGO_OBJECT_ID) already exists.
	at com.torodb.backend.AbstractErrorHandler.handleUserException(AbstractErrorHandler.java:91) ~[backend-common-0.50.2.jar:?]
	at com.torodb.backend.AbstractErrorHandler.handleException(AbstractErrorHandler.java:66) ~[backend-common-0.50.2.jar:?]
	at com.torodb.backend.SqlHelper.executeUpdate(SqlHelper.java:101) ~[backend-common-0.50.2.jar:?]

The application still seems to work, and all rows are successfully imported (although we're still seeing way more tables than we should.) Notably, there are no duplicate mongo ids in the collections we're importing or in the resulting tables in postgres. However, perhaps related, the postgres ids (_id_x) appear to malformed, as if the original mongo id has not been migrated as a string (e.g. W3]J\272zQs:\254O{ or W'\236\225\272zQg\253a8\326).

Do you have any information on this error? Google is not offering any insight.

Thanks!

[Matteo Melli]

Hi @mr-sk,

This is a normal "error" message that is reported by the backend (postgresql) due to the way we import data using parallel connections (when two column are created at the same time for two different documents with similar structure). You can ignore the message. By the way are you running a development version built from source code? - That would explain why such message appears.

On the other side, the mongo object id are stored as bytea in postgresql and contains essentially the same value but it is represented with a different format.

Cheers