This component lets you publish and subscribe to Kafka and GCP PubSub events.
import lightning as L
from lit_kafka import KafkaRootFlow
if __name__ == "__main__":
app = L.LightningApp(
KafkaRootFlow("kafka-test", bootstrap_servers="localhost:9092", num_partitions=2)
)
Use these instructions to install:
git clone https://github.com/PyTorchLightning/lightning-pubsub.git
cd lightning-lit_kafka
pip install -r requirements.txt
pip install -e .