yougov / mongo-connector

MongoDB data stream pipeline tools by YouGov (adopted from MongoDB)

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Mongo-connector seems not to work with transactions - doesn't sync with elasticsearch

AmrAdelKhalil opened this issue · comments

Hello,
I have mongo with two replica-sets and I'm using mongo-connector to sync the data to elasticsearch, It was working fine till I started to use Transactions in my code.
When I logged the mongo-connector service it was showing regular output like:

2021-07-13 19:35:30,625 [INFO] mongo_connector.oplog_manager:76 - OplogThread for replica set 'rs0' is up to date with the oplog.

and when I run the script which include the transaction, the logs showed this:

2021-07-13 19:35:41,166 [INFO] elasticsearch:81 - POST http://host:9200/_refresh [status:200 request:0.004s]

I tried to test this by inserting another document without the transaction and it showed:

2021-07-13 19:21:59,465 [INFO] elasticsearch:81 - POST http://marketplace2-elasticsearch-client:9200/_bulk [status:200 request:0.012s]

Now, what I can see is that the transaction is going somehow wrong with mongo-connector.
Can anyone help me with that?

mongodb version: 4.0
mongo-connector version: latest & using: mongo-connector[elastic5]