yougov / mongo-connector

MongoDB data stream pipeline tools by YouGov (adopted from MongoDB)

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Missing documents while connecting Mongodb and elasticsearch via mongo-connector

FatehAlc opened this issue · comments

I am trying to transfer documents from MongoDB 4.4.5 to Elasticsearch7.12 via mongo-connector 3.1.1

I start by using:

sudo mongo-connector -m 127.0.0.1:27017 -t 127.0.0.1:9200 -d elastic2_doc_manager -n Data.myData --continue-on-error --batch-size=1000 --only-dump --batch-size=1000 --no-dump --tz-aware

at the beginning of the log, I can see the total number of the documents in MongoDB which is 454
image

when the process is done I get a message saying
[DEBUG] elasticsearch:266 - < {"_shards":{"total":44,"successful":29,"failed":0}}

and in Elasticsearch index I can only see 268 as in the images shows

image

any idea what could be the issue here?

This issue is Releated to reaching the max number for fields's mapping on Elastivsearch
to Solve this issue increase this limit on Elasticsearch's index via

PUT my_data_index/_settings
{
  "index.mapping.total_fields.limit": 20000

}