Unable to use message compression with kafka outputplugin

[sisteinmetz]

Hello,

Due to this issue: fluent/fluent-bit#5957
I rebuild fluentbit from source from master (i have to do it from master due to this issue: fluent/fluent-bit#5300)

i followed this doc: https://docs.fluentbit.io/manual/installation/macos
I do th cmake:
sudo cmake -DFLB_OUT_KAFKA=Yes -DCMAKE_INSTALL_PREFIX=/opt/fluent-bit ../

and then :
sudo make -j 16

and then:
sudo make install

Everything seems ok

i used this conf:

[INPUT]
name dummy
tag dummy.local

[OUTPUT]
name stdout
match *

[OUTPUT]
Name kafka
Match *
Brokers localhost:9092
Topics dummy-gzip
rdkafka.client.id gzip
rdkafka.compression.type gzip
rdkafka.compression.codec gzip

and i start fluentbit:
ssteinmetz@XXXXXX build % sudo /opt/fluent-bit/bin/fluent-bit -c ~/Documents/Siemens/fluent-bit.conf
Fluent Bit v2.0.0

• Copyright (C) 2015-2022 The Fluent Bit Authors
• Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
• https://fluentbit.io

[2022/10/13 16:53:21] [ info] [fluent bit] version=2.0.0, commit=, pid=61073
[2022/10/13 16:53:21] [ info] [storage] ver=1.3.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2022/10/13 16:53:21] [ info] [cmetrics] version=0.5.2
[2022/10/13 16:53:21] [ info] [input:dummy:dummy.0] initializing
[2022/10/13 16:53:21] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2022/10/13 16:53:21] [ info] [output:stdout:stdout.0] worker #0 started
[2022/10/13 16:53:21] [ info] [output:kafka:kafka.1] brokers='localhost:9092' topics='dummy-gzip'
[2022/10/13 16:53:21] [ info] [sp] stream processor started
[0] dummy.local: [1665672802.143611000, {"message"=>"dummy"}]
[0] dummy.local: [1665672803.142710000, {"message"=>"dummy"}]
[0] dummy.local: [1665672804.142563000, {"message"=>"dummy"}]

All messages arrive in dummy-gzip topic.
I use kafka 3.1.0.
On this topic compression.codec is producer.

When i check the message compression:

ssteinmetz@Yoga460-90KRVPB Documents % ./confluent-7.2.2/bin/kafka-run-class kafka.tools.DumpLogSegments --files "/Users/ssteinmetz/Library/Application Support/conduktor/jumpstart/3fbc7d57-46de-49b1-bfb7-7eeae91d17ca/data/kafka/dummy-gzip-0/00000000000000000000.log" --print-data-log | grep compresscodec

baseOffset: 0 lastOffset: 0 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 0 isTransactional: false isControl: false deleteHorizonMs: OptionalLong.empty position: 0 CreateTime: 1665672803143 size: 118 magic: 2 compresscodec: none crc: 1422198876 isvalid: true
baseOffset: 1 lastOffset: 1 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 0 isTransactional: false isControl: false deleteHorizonMs: OptionalLong.empty position: 118 CreateTime: 1665672804142 size: 117 magic: 2 compresscodec: none crc: 3493017080 isvalid: true
baseOffset: 2 lastOffset: 2 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 0 isTransactional: false isControl: false deleteHorizonMs: OptionalLong.empty position: 235 CreateTime: 1665672805143 size: 118 magic: 2 compresscodec: none crc: 1915063210 isvalid: true

All compression codec are none

I have the same beaviour if i use rdkafka.compression.codec lz4, zstd, ....

[sisteinmetz]

I closed this issue toi re-open it in fluent-bit repo