The openresty kafka client library that encapsulates kafka-go
# download and build
cd /opt
git clone https://github.com/kingluo/lua-resty-ffi-kafka
cd lua-resty-ffi-kafka
make
# start kafka container
cd demo
docker compose up -d
# create topic
python3 create-topic.py
# start nginx
mkdir logs
LD_LIBRARY_PATH=/opt/lua-resty-ffi-kafka/:/usr/local/lib/lua/5.1 \
nginx -p $PWD -c nginx.conf
# test sync send
curl http://localhsot:20000/send?cnt=1
lua-resty-kafka has below issues:
- the author does not maintain the project positively anymore
- features missing
- kafka version compatibility
- compression
- consumer group
- reconnections
- more SASL methods
- ...
- bugs
- no necessary metadata refresh ahead of send/receive
- code reentrancy problem (e.g. concurrent sync sends)
- ...
kafka-go is a popular and active kafka client library.
Why not encapsulate it so that we could reuse it in openresty?
lua-resty-ffi provides an efficient and generic API to do hybrid programming in openresty with mainstream languages (Go, Python, Java, Rust, Nodejs).
lua-resty-ffi-kafka = lua-resty-ffi + kafka-go
For the time being, lua-resty-ffi-kafka is 80% of the performance of lua-resty-kafka.