hpgrahsl kafka-connect-mongodb and geojson data

Question

hpgrahsl kafka-connect-mongodb and geojson data

opistara opened this issue 5 years ago · comments

Hello Hans-Peter,
I'm using your kafka-connect-mongodb v 1.3.1, with Kafka 0.10.1 and schema registry (confluent) ver 3.1

I also use Nifi 1.5.0 to ingest data and publish it in Kafka topic by publishKafkaRcord processor.

Now if i ingest, a flat, csv file associated to a simple avro schema, everything is ok.
Infact i ingest CSV data, procces single record, publish on Kafka topic, and then mongodb sink consume from topic and save on mongoDB collection. Wonderful!!!

But if i ingest geojson file, my pipeline breaks on your mongoDB sink.

I give you more information about my geojson data processing.

1)To simplify i ingest a geojson with a single record
[ {
"type" : "Feature",
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
18.183730244636536,
40.35109032638699
],
[
18.183746337890625,
40.351016739174426
],
[
18.183730244636536,
40.35109032638699
]
]
]
},
"properties": {
"name": "Z-04",
"description": "parking 1,30 €/hour"
}
}]

this the avro schema that i have regitered on schema-registry

{"type": "record", "name": "piano_sosta_tariffata_lecce", "namespace": "opendata.piano_sosta-le", "fields": [
{"name" : "type", "type" : "string"},
{"name" : "geometry", "type" : {
"type" : "record", "name" : "_geometry", "fields" : [
{"name": "type", "type": "string"},
{"name" : "coordinates", "type" : {
"type": "array",
"items": {
"type": "array",
"items":
{"type": "array",
"items" : "double"
}
}
}
}
]
}
},
{"name" : "properties", "type" :
{
"type" : "record", "name" : "_properties", "fields" : [
{"name": "name", "type": "string"},
{"name": "description", "type": "string"}
]
}
},
{"name": "ingestion_date", "type":
{"type": "int", "logicalType":"date"}
},
{"name": "ingestion_timestamp", "type":
{ "type": "long", "logicalType": "timestamp-millis"}
}
]
}

I have also used an on-line avro validator (https://json-schema-validator.herokuapp.com/avro.jsp) to check the avro schema correctness.

I ingest and publish it on kafka topic, and using "kafka-avro-console-consumer" tool with following command
sudo bin/kafka-avro-console-consumer --zookeeper hdpmaster1.it:2181 --topic raw_OD_LE_piano_sosta_tariffata --from-beginning
I consume and can see the record
{"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[18.183730244636536,40.35109032638699],[18.183746337890625,40.351016739174426],[18.183730244636536,40.35109032638699]]]},"properties":{"name":"","description":""},"ingestion_date":0,"ingestion_timestamp":1570201210514}
At this point, even WorkerSinkTask mongoDB, try to consume the same record on Kafka topic, but it has problem during deserialization, i think.
This the error that i found on confluent-kafka-connect log:

Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: Oct 09 13:37:09 hdpmaster1 connect-distributed.sh[27228]: [2019-10-09 13:37:09,515] ERROR Task mongodb_sink_OD_LE_piano_sosta_tariffata-0 threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerS
org.apache.kafka.connect.errors.DataException: error while processing field geometry
at at.grahsl.kafka.connect.mongodb.converter.AvroJsonSchemafulRecordConverter.processField(AvroJsonSchemafulRecordConverter.java:131)
at at.grahsl.kafka.connect.mongodb.converter.AvroJsonSchemafulRecordConverter.lambda$toBsonDoc$0(AvroJsonSchemafulRecordConverter.java:92)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)
at at.grahsl.kafka.connect.mongodb.converter.AvroJsonSchemafulRecordConverter.toBsonDoc(AvroJsonSchemafulRecordConverter.java:92)
at at.grahsl.kafka.connect.mongodb.converter.AvroJsonSchemafulRecordConverter.convert(AvroJsonSchemafulRecordConverter.java:78)
at at.grahsl.kafka.connect.mongodb.converter.SinkConverter.convert(SinkConverter.java:50)
at at.grahsl.kafka.connect.mongodb.MongoDbSinkTask.lambda$buildWriteModel$3(MongoDbSinkTask.java:213)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at at.grahsl.kafka.connect.mongodb.MongoDbSinkTask.buildWriteModel(MongoDbSinkTask.java:212)
at at.grahsl.kafka.connect.mongodb.MongoDbSinkTask.processSinkRecords(MongoDbSinkTask.java:143)
at at.grahsl.kafka.connect.mongodb.MongoDbSinkTask.lambda$null$0(MongoDbSinkTask.java:118)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at at.grahsl.kafka.connect.mongodb.MongoDbSinkTask.lambda$put$1(MongoDbSinkTask.java:117)
at java.util.HashMap.forEach(HashMap.java:1289)
at at.grahsl.kafka.connect.mongodb.MongoDbSinkTask.put(MongoDbSinkTask.java:112)
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:384)
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:240)
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:172)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:143)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:140)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:175)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.connect.errors.DataException: error while processing field coordinates
at at.grahsl.kafka.connect.mongodb.converter.AvroJsonSchemafulRecordConverter.processField(AvroJsonSchemafulRecordConverter.java:131)
at at.grahsl.kafka.connect.mongodb.converter.AvroJsonSchemafulRecordConverter.lambda$toBsonDoc$0(AvroJsonSchemafulRecordConverter.java:92)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)
at at.grahsl.kafka.connect.mongodb.converter.AvroJsonSchemafulRecordConverter.toBsonDoc(AvroJsonSchemafulRecordConverter.java:92)
at at.grahsl.kafka.connect.mongodb.converter.AvroJsonSchemafulRecordConverter.handleStructField(AvroJsonSchemafulRecordConverter.java:178)
at at.grahsl.kafka.connect.mongodb.converter.AvroJsonSchemafulRecordConverter.processField(AvroJsonSchemafulRecordConverter.java:119)
... 26 more
Caused by: org.apache.kafka.connect.errors.DataException: Cannot list fields on non-struct type
at org.apache.kafka.connect.data.ConnectSchema.fields(ConnectSchema.java:177)
at at.grahsl.kafka.connect.mongodb.converter.AvroJsonSchemafulRecordConverter.toBsonDoc(AvroJsonSchemafulRecordConverter.java:92)
at at.grahsl.kafka.connect.mongodb.converter.AvroJsonSchemafulRecordConverter.handleArrayField(AvroJsonSchemafulRecordConverter.java:168)
at at.grahsl.kafka.connect.mongodb.converter.AvroJsonSchemafulRecordConverter.processField(AvroJsonSchemafulRecordConverter.java:122)
... 32 more
[2019-10-09 13:37:09,516] ERROR Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerSinkTask:405)
[2019-10-09 13:37:09,516] INFO WorkerSinkTask{id=mongodb_sink_OD_LE_piano_sosta_tariffata-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSinkTask:262)
[2019-10-09 13:37:09,519] ERROR Task mongodb_sink_OD_LE_piano_sosta_tariffata-0 threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerT
org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:406)
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:240)
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:172)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:143)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:140)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:175)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-10-09 13:37:09,519] ERROR Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:143)
[2019-10-09 13:37:09,519] INFO stopping MongoDB sink task (at.grahsl.kafka.connect.mongodb.MongoDbSinkTask:258)

I suspect that the "array of array" presence in my geojson data creates deserialization problem to the WorkerSinkTask mongoDB.
Is there some aspect I am missing? Can you help me to understand what is the problem
and what workaround adopt to overcome it.

Cheers and thx.

Orazio

Hans-Peter Grahsl · Answer 1 · Fri Oct 18 2019 14:46:29 GMT+0800 (China Standard Time)

@opistara thx for reaching out. glad to hear you like this project and find it useful!

Based on your detailed issue description including logs it looks very much that the actual problem is indeed related to a multi-level nesting of arrays. I will further investigate to be sure and report back. Cannot promise when there will be a fix for this but it shouldn't take too long.

Hans-Peter Grahsl · Answer 2 · Fri Nov 29 2019 21:12:18 GMT+0800 (China Standard Time)

@opistara you may try the connector based on this branch / snapshot

Hans-Peter Grahsl · Answer 3 · Fri Nov 29 2019 21:43:49 GMT+0800 (China Standard Time)

@opistara it's merged now. so just build the latest from master branch and give it a try.

Orazio · Answer 4 · Mon Dec 02 2019 17:18:26 GMT+0800 (China Standard Time)

@hpgrahsl thanks a lot.

Hans-Peter Grahsl · Answer 5 · Sat Dec 14 2019 05:16:04 GMT+0800 (China Standard Time)

@opistara does it work for your GeoJSON use case now with the provided fix? would be nice to know :)