Cannot read kafka messages with local docker setup

[Ernest Bednarczyk]

Hi, I'm trying to create test / dev setup with such docker-compose config:

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    networks:
      - broker-kafka
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
  kafka:
    image: confluentinc/cp-kafka:latest
    networks:
      - broker-kafka
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
  kafdrop:
    image: obsidiandynamics/kafdrop:latest
    networks:
      - broker-kafka
    ports:
      - 19000:9000
    environment:
      KAFKA_BROKERCONNECT: kafka:29092
  arroyo:
    image: ghcr.io/arroyosystems/arroyo-single:latest
    networks:
      - broker-kafka
    ports:
      - 8000:8000
networks:
  broker-kafka:
    driver: bridge

When I'm creating those containers and try to create source connection to kafka (using instractions in https://doc.arroyo.dev/tutorial/kafka), I'm receiving response:

Connected to Kafka
Fetched topic metadata
Waiting for messages
No messages received from Kafka within 30 seconds

In the same time I'm producing messages to the topic, what I can observe using Kafdrop.

Please advise, what can be a solution for that.

[Micah Wylde]

Thanks for checking out Arroyo!

This worked for me when I tried your docker compose script.

The steps I followed:

1. Ran docker-compose up, starting the services
2. Opened the kafdrop UI at http://localhost:19000/ and created a topic my-topic
3. Exec'd into the kafka container and produced a message to the topic:

$ docker exec -it bcba7d382f4e /bin/bash
[appuser@bcba7d382f4e ~]$ kafka-console-producer --bootstrap-server localhost:9092 --topic my-topic
>test

4. Opened Arroyo at http://localhost:8000/ and navigated to connections -> create connection -> kafka
5. Created a new cluster:
   
6. Configured the topic:
   
7. Set the format to raw string
8. Gave it a name 'mytopic'
9. Clicked test connection:
   

Do you mind trying again with those steps? We may be able to help you more efficiently in discord as well.

[Micah Wylde]

Closing for now — feel free to re-open or reach out if you continue to have issues!