scylladb / kafka-connect-scylladb

Kafka Connect Scylladb Sink

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

avro deserializer error

dhgokul opened this issue · comments

Using redpanda as a broker
Using Redpanda's default schema registry

    schema ="""    {
        "namespace": "confluent.io.examples.serialization.avro",
        "name": "user2",
        "type": "record",
        "fields": [
            {"name": "id", "type": "string"},
            {"name": "emailid", "type": "string"},
            {"name": "ip", "type": "string"},
            {"name": "browser", "type": "string"},
            {"name": "segment", "type": "string"}
        ]
    }"""

Actual payload:
{"id":"123","emailid":"user1@email.com","ip":"192.112.222.183","browser":"Firefox-57.0.4","segment":"home"}

connector properties:

bootstrap.servers=redpanda_server0:9092,redpanda_server1:9092,redpanda_server3:9092
key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://redpanda_server0:8081
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://redpanda_server0:8081
offset.storage.file.filename=/tmp/connect.offsets 
plugin.path=target/components/packages/

sink properties:

name=scylladb-sink-connector2
connector.class=io.connect.scylladb.ScyllaDbSinkConnector
tasks.max=1
topics=topic1
scylladb.contact.points=scylla1,scylla2,scylla3
scylladb.port=9042
scylladb.keyspace=redpanda
key.converter=org.apache.kafka.connect.storage.StringConverter
key.converter.schema.registry.url=http://redpanda_server0:8081
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://redpanda_server0:8081
key.converter.schemas.enable=true
value.converter.schemas.enable=true
key.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
value.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
transforms=createKey
transforms.createKey.fields=id
transforms.createKey.type=org.apache.kafka.connect.transforms.ValueToKey

facing deserializer error:

Caused by: org.apache.kafka.connect.errors.DataException: Failed to deserialize data for topic topic1 to Avro: 
Caused by: org.apache.kafka.common.errors.SerializationException: Error retrieving Avro value schema version for id 7

Since facing an error on avro now using JSON method:

connector:

bootstrap.servers=redpanda_server0:9092,redpanda_server1:9092,redpanda_server3:9092

key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
offset.storage.file.filename=/tmp/connect.offsets 
plugin.path=target/components/packages/

sink property:

name=scylladb-sink-connector
connector.class =io.connect.scylladb.ScyllaDbSinkConnector
tasks.max= 1
topics =topic1
scylladb.contact.points=scylla1,scylla2,scylla3
scylladb.port=9042
scylladb.keyspace=redpanda
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true
transforms=createKey
transforms.createKey.fields=id
transforms.createKey.type=org.apache.kafka.connect.transforms.ValueToKey

Now producing message on redpanda as:
{"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"},{"type":"string","optional":false,"field":"emailid"},{"type":"string","optional":false,"field":"ip"},{"type":"string","optional":false,"field":"browser"},{"type":"string","optional":false,"field":"segment"}]},"payload":{"id":"9999993","emailid":"user1@email.com","ip":"192.112.222.183","browser":"Firefox-57.0.4","segment":"purchase"}}

But this method is not efficient in case producing large numbers of messages.

So looking for Avro format but showing deserializer error. Are there any changes that have to do with sink or connector properties?