kafka-ops / julie

A solution to help you build automation and gitops in your Apache Kafka deployments. The Kafka gitops!

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Use different strategies for the key and the value in the same topic

vascoferraz opened this issue · comments

Hi everyone :)

I read the documentation and I did not find any way to use different strategies, that is, TopicNameStrategy, RecordNameStrategy or TopicRecordNameStrategy, for the key and the value in the same topic.

I also tried the following configurations but them failed while deploying:

First attempt - Add two topics entries (one for the value and another for the key)

context: "XX"
source: "yyyyy"
projects:
  - name: "my-project"
    topics:
      - name: "xxx-customer-v1"
        dataType: "avro"
        subject.name.strategy: "RecordNameStrategy"
        schemas:
            value.schema.file: "schemas/xxx-new_customer-v1-value.avsc"
            value.record.type: "com.xxx.customers.NewCustomer"
      - name: "xxx-customer-v1"
        dataType: "avro"
        subject.name.strategy: "TopicNameStrategy"
        schemas:
          - key.schema.file: "schemas/xxx-customer-v1-key.avsc"
            key.record.type: "java.lang.String"
        config:
          replication.factor: "3"
          num.partitions: "3"
          compression.type: "gzip"

Exception in thread "main" java.lang.IllegalArgumentException: com.purbon.kafka.topology.exceptions.ValidationException: Missing required value.schema.file on schemas for topic xxx-customer-v1

Second attempt: Duplicate the topic entry and set RecordNameStrategy for the value (first entry) and TopicNameStrategy for the key (second entry).

context: "XX"
source: "yyyyy"
projects:
  - name: "my-project"
    topics:
      - name: "xxx-customer-v1"
        dataType: "avro"
        subject.name.strategy: "RecordNameStrategy"
        schemas:
          - key.schema.file: "schemas/xxx-customer-v1-key.avsc"
            key.record.type: "java.lang.String"
            value.schema.file: "schemas/xxx-new_customer-v1-value.avsc"
            value.record.type: "com.xxx.customers.NewCustomer"
      - name: "xxx-customer-v1"
        dataType: "avro"
        subject.name.strategy: "TopicNameStrategy"
        schemas:
          - key.schema.file: "schemas/xxx-customer-v1-key.avsc"
            key.record.type: "java.lang.String"
            value.schema.file: "schemas/xxx-new_customer-v1-value.avsc"
            value.record.type: "com.xxx.customers.NewCustomer"
        config:
          replication.factor: "3"
          num.partitions: "3"
          compression.type: "gzip"

Exception in thread "main" java.lang.IllegalStateException: Duplicate key xxx-customer-v1