Missing documents while connecting Mongodb and elasticsearch via mongo-connector
FatehAlc opened this issue · comments
I am trying to transfer documents from MongoDB 4.4.5 to Elasticsearch7.12 via mongo-connector 3.1.1
I start by using:
sudo mongo-connector -m 127.0.0.1:27017 -t 127.0.0.1:9200 -d elastic2_doc_manager -n Data.myData --continue-on-error --batch-size=1000 --only-dump --batch-size=1000 --no-dump --tz-aware
at the beginning of the log, I can see the total number of the documents in MongoDB which is 454
when the process is done I get a message saying
[DEBUG] elasticsearch:266 - < {"_shards":{"total":44,"successful":29,"failed":0}}
and in Elasticsearch index I can only see 268 as in the images shows
any idea what could be the issue here?
This issue is Releated to reaching the max number for fields's mapping on Elastivsearch
to Solve this issue increase this limit on Elasticsearch's index via
PUT my_data_index/_settings
{
"index.mapping.total_fields.limit": 20000
}