rajalingams / kafka-connect-smts-nested-fields

Custom Kafka Connect SMT to set the Key and/or Header from Value using Json path

Collection of Kafka Connect SMTs to set the key and/or header from the value by using JsonPath (Supports nested values).

NestedValueToKey

Custom Kafka Connect SMT to set the key based on the value using Json path. The raw value of the field will be replaced as the message key. Use StringConverter for Key.

Properties:

Name	Description	Type	Default	Required	Importance
field	Field name to be used as key, use json path	String	null	true	High
keyOnError	Constant value to be used as key when there is an error	String	null	false	Low
errorMessageField	Field name to be used when printing errors, use json path	String	null	false	Low

Example:

Message Value:

{
  "name": "apple",
  "category": "fruit",
  "subcategory": "HoneyCrisp",
  "location": {
    "aisle": 10,
    "row": 3
  },
  "price": 1.59
}

Config:

transforms=nestedValueToKey
transforms.nestedValueToKey.type=io.confluent.connect.custom.transforms.NestedValueToKey
transforms.nestedValueToKey.field="$.name"
transforms.nestedValueToKey.keyOnError="unnamed"
transforms.nestedValueToKey.errorMessageField="$.category"

Key generated by SMT:

In case the field is missing or empty in the message value, the key will be "unnamed"

apple

---

NestedValueToKeys

Custom Kafka Connect SMT to set the message Key based on the value using Json path. Key will be replaced by a map of field name to value.

Properties:

Name	Description	Type	Default	Required	Importance
keyFieldMapping	Map of key field name to json path in the message value. eg: field1:jsonpath1,field2:jsonpath2..	String	null	true	High

Example:

Message Value:

{
  "name": "apple",
  "category": "fruit",
  "subcategory": "HoneyCrisp",
  "location": {
    "aisle": 10,
    "row": 3
  },
  "price": 1.59
}

Config:

transforms=nestedValueToKeys
transforms.nestedValueToKeys.type=io.confluent.connect.custom.transforms.NestedValueToKeys
transforms.nestedValueToKeys.keyFieldMapping="item:$.name,aisle:$.location.aisle"

Key generated by SMT:

In case the field name is missing or empty in the message value, the field value will be null

{
  "item": "apple",
  "aisle": 10
}

---

NestedValueToHeader

Custom Kafka Connect SMT to set the message Header based on the value using Json path. Header will be a map of fieldname to the value. It will append values to the header if it is already present.

Properties:

Name	Description	Type	Default	Required	Importance
headerFieldMapping	Map of header field name to json path in the message value. eg: field1:jsonpath1,field2:jsonpath2..	String	null	true	High

Example:

Message Value:

{
  "name": "apple",
  "category": "fruit",
  "subcategory": "HoneyCrisp",
  "location": {
    "aisle": 10,
    "row": 3
  },
  "price": 1.59
}

Config:

transforms=nestedValueToHeader
transforms.nestedValueToHeader.type=io.confluent.connect.custom.transforms.NestedValueToHeader
transforms.nestedValueToHeader.headerFieldMapping="item:$.name,aisle:$.location.aisle"

Header generated by SMT:

In case the field name is missing or empty in the message value, the field value will be null

[
  {
    "key": "item",
    "stringValue": "apple"
  },
  {
    "key": "aisle",
    "stringValue": "10"
  }
]

---

NestedValueToKeyAndHeader

Custom Kafka Connect SMT to set the key and header based on the value(s) using Json path. The raw value of the field will be replaced as the message key. Header will be a map of fieldname to the value. It will append values to the header if it is already present. Use StringConverter for Key.

Properties:

Name	Description	Type	Default	Required	Importance
field	Field name to be used as key, use json path	String	null	true	High
keyOnError	Constant value to be used as key when there is an error	String	null	false	Low
errorMessageField	Field name to be used when printing errors, use json path	String	null	false	Low
headerFieldMapping	Map of header field name to json path in the message value. eg: field1:jsonpath1,field2:jsonpath2..	String	null	false	Medium

Example:

Message Value:

{
  "name": "apple",
  "category": "fruit",
  "subcategory": "HoneyCrisp",
  "location": {
    "aisle": 10,
    "row": 3
  },
  "price": 1.59
}

Config:

transforms=nestedValueToKeyAndHeader
transforms.nestedValueToKeyAndHeader.type=io.confluent.connect.custom.transforms.NestedValueToKeyAndHeader
transforms.nestedValueToKeyAndHeader.field="$.name"
transforms.nestedValueToKeyAndHeader.keyOnError="unnamed"
transforms.nestedValueToKeyAndHeader.errorMessageField="$.category"
transforms.nestedValueToKeyAndHeader.headerFieldMapping="item:$.name,aisle:$.location.aisle"

Key generated by SMT:

In case the field name is missing or empty in the message value, the key will be "unnamed"

apple

Header generated by SMT:

In case the field name is missing or empty in the message value, the field value will be null

[
  {
    "key": "item",
    "stringValue": "apple"
  },
  {
    "key": "aisle",
    "stringValue": "10"
  }
]

---

NestedValueToKeysAndHeader

Custom Kafka Connect SMT to set the message Key based on the value using Json path. Key will be replaced by a map of field name to value. Header will be a map of fieldname to the value. It will append values to the header if it is already present.

Properties:

Name	Description	Type	Default	Required	Importance
keyFieldMapping	Map of key field name to json path in the message value. eg: field1:jsonpath1,field2:jsonpath2..	String	null	true	High
headerFieldMapping	Map of header field name to json path in the message value. eg: field1:jsonpath1,field2:jsonpath2..	String	null	true	High

Example:

Message Value:

{
  "name": "apple",
  "category": "fruit",
  "subcategory": "HoneyCrisp",
  "location": {
    "aisle": 10,
    "row": 3
  },
  "price": 1.59
}

Config:

transforms=nestedValueToKeysAndHeader
transforms.nestedValueToKeysAndHeader.type=io.confluent.connect.custom.transforms.NestedValueToKeysAndHeader
transforms.nestedValueToKeysAndHeader.keyFieldMapping="item:$.name,aisle:$.location.aisle"
transforms.nestedValueToKeysAndHeader.headerFieldMapping="type:$.category,subtype:$.subCategory"

Key generated by SMT:

In case the field name is missing or empty in the message value, the field value will be null

{
  "item": "apple",
  "aisle": 10
}

Header generated by SMT:

In case the field name is missing or empty in the message value, the field value will be null

[
  {
    "key": "type",
    "stringValue": "fruit"
  },
  {
    "key": "row",
    "stringValue": "3"
  }
]

---

Note

SMTs are applicable only if the message value has a schema or if it is a map.