Unable to explicitly set no key in kafka franz output

[David Shrive]

Previously when using the kafka output we used the following interpolation to set the kafka key.

      key: ${!meta("kafka_key").not_null().catch("")}

The behaviour here was to use the kafka_key added at input if its available but if not, don't error and don't set a key.

However when using the exact same config on the kafka_franz output the behaviour changed as the 0 length string was instead used as the key, causing everything to go to the same partition. I also tried catch(null) which did the same thing.

[Ashley Jeffs]

@davidshrive you could do ${!meta("kafka_key").not_null().catch(uuid_v4())}

[David Shrive]

Yes, our solution for the moment is to always set a key. I do think it would be good to have the option to set no key though.

[Ashley Jeffs]

I actually think having an empty key route to the same partition might be less confusing than what the original kafka output is doing, as the latter must be generating a random hash when the key is empty which is a hidden magic behaviour.

If you don't want to partition based on the key itself there's also the option of alternative partitioning schemes: https://www.benthos.dev/docs/components/outputs/kafka_franz#partitioner.