[Bug] MQBrokerException: CODE: 206 DESC: the consumer group[consumer-xxx] not online BROKER: xxx:10911 For more information

[xiaolyuh123]

Before Creating the Bug Report

• I found a bug, not just asking a question, which should be created in GitHub Discussions.

• I have searched the GitHub Issues and GitHub Discussions of this repository and believe that this is not a duplicate.

• I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.

Runtime platform environment

k8s 1.28.1

RocketMQ version

镜像版本：
apache/rocketmq:5.3.0
apacherocketmq/rocketmq-dashboard:1.0.0

JDK Version

No response

Describe the Bug

我在k8s环境中，通过helm 部署了 rocketmq5.3 发送和消费消息都正常（走代理），但是在rocketmq-dashboard 的控制台里面查看消费者终端时报错 ：
rocketmq-console MQBrokerException: CODE: 206 DESC: the consumer group[consumer-xxx] not online BROKER: 10.244.135.137:10911 For more information
查看生产者是报错：
org.apache.rocketmq.client.exception.MQBrokerException: CODE: 1 DESC: the producer group[xxx] not exist BROKER: 10.244.135.135:10911 For more information

Steps to Reproduce

登录dashbord，点击【消费者】，点击订阅组对应的【终端】按钮会报错：
rocketmq-console MQBrokerException: CODE: 206 DESC: the consumer group[consumer-xxx] not online BROKER: 10.244.135.137:10911 For more information

点击【生产者】，选择topic后点击【搜索】按钮会报错：
org.apache.rocketmq.client.exception.MQBrokerException: CODE: 1 DESC: the producer group[xxx] not exist BROKER: 10.244.135.135:10911 For more information

What Did You Expect to See?

我希望能正常看到生产者的消费者

What Did You See Instead?

错误提示

Additional Context

No response

[彭镜肇]

我仔细看了你的问题，可能是因为 Dashboard 和 RocketMQ 集群之间的通信存在问题或状态同步异常。下面是我的一些建议：

可能存在的问题：

1. 消费者和生产者状态未正确同步到 Broker
   • 消费者组未在线可能是由于消费者没有发送心跳到 Broker 或者 Dashboard 无法正确访问 Broker 的状态数据。
   • 生产者组不存在可能是因为 Dashboard 无法拉取到生产者的状态信息。
2. RocketMQ Dashboard 和 RocketMQ 集群通信不正常
   • Dashboard 依赖于 NameServer 和 Broker 的状态信息，可能存在网络通信问题导致 Dashboard 无法获取到正确的状态。
3. RocketMQ 集群和 Kubernetes 网络设置问题
   • 部署在 Kubernetes 中时，可能存在外部访问或 DNS 配置问题导致 Dashboard 和 Broker 的通信失败。典型问题包括：
     • NameServer 和 Broker 的地址解析问题（可能返回了内网 IP）。
     • 集群的 brokerIP1 或 advertisedAddr 配置不正确。
     • Dashboard 的 ROCKETMQ_NAMESRV_ADDR 配置未正确指向 NameServer。
4. Broker 不支持拉取消费者/生产者组状态
   • RocketMQ 的某些版本对生产者和消费者组状态的检测依赖于消费者和生产者的心跳机制（默认情况下 RocketMQ 只有在心跳发送后才会记录状态）。

解决方案：

1. 检查并确认网络配置

   • 确认 Kubernetes 中 RocketMQ 的部署是否正常。
   • 确认 NameServer 和 Broker 的通信是否正常：
     kubectl exec -it -- curl http://:9876
   • 确认 Dashboard 的 ROCKETMQ_NAMESRV_ADDR 配置是否正确：
     • Dashboard 的容器中需要配置 RocketMQ NameServer 的地址，环境变量为 ROCKETMQ_NAMESRV_ADDR。
     • 检查 Dashboard 的配置：
       kubectl describe pod
       确保 ROCKETMQ_NAMESRV_ADDR 指向正确的 NameServer 地址（如 rocketmq-nameserver.default.svc.cluster.local:9876）。

2. 检查 Broker 配置

   • 确保 Broker 的 brokerIP1 和 advertisedAddr 配置正确，指向集群的访问地址（而不是容器的私有 IP 地址）。
     • 在 broker.conf 中设置：
       brokerIP1=<节点可访问的 IP>
       brokerClusterName=DefaultCluster
     • 如果使用 Helm，可以通过 values.yaml 配置 brokerIP1 和 advertisedAddr。
   • 检查 broker.conf 配置是否完整，特别是以下关键配置：
     namesrvAddr=<NameServer地址，例如 rocketmq-nameserver.default.svc.cluster.local:9876>
     listenPort=10911
   • 重启 Broker 后，观察 Broker 和 NameServer 是否成功注册。

3. 检查消费者和生产者的状态

   • 消费者组状态：
     • 消费者组状态需要消费者发送心跳到 Broker。如果消费者没有正常启动或长时间未发送心跳，会导致消费者组状态显示为不在线。
     • 确保消费者应用正常启动并消费消息，可以通过日志查看消费者是否正常工作。
   • 生产者状态：
     • RocketMQ 默认不存储生产者组的静态信息，只有生产者向 Broker 发送消息之后，Broker 才会记录生产者组信息。
     • 确保生产者应用向指定的 Topic 发送消息，并检查生产者的日志是否有异常。

4. 拉取消费者和生产者的状态

   • 使用命令行工具手动拉取消费者/生产者的状态，以排除 Dashboard 的问题：
     • 查询消费者组状态：
       kubectl exec -it -- bash
       sh mqadmin consumerStatus -n <nameserver地址> -g
     • 查询生产者组状态：
       sh mqadmin producerConnection -n <nameserver地址> -g

5. 检查 Dashboard 的版本适配问题

   • 您使用的 Dashboard 是 1.0.0，而 Broker 是 5.3.0。Dashboard 的版本较老，可能存在兼容性问题。
   • 尝试升级 Dashboard 到与 RocketMQ 5.3.0 兼容的版本（如 dashboard 1.1.0 或更高）。

6. 检查日志

   • NameServer 日志：检查 NameServer 是否成功注册 Broker 和 Consumer 的信息：
     kubectl logs
   • Broker 日志：检查 Broker 是否接收到来自消费者和生产者的心跳：
     kubectl logs
   • Dashboard 日志：检查 Dashboard 是否能正常访问 NameServer 和 Broker：
     kubectl logs

7. 配置 Dashboard 的 Debug 模式

   • 如果问题仍未解决，可以尝试将 RocketMQ Dashboard 设置为调试模式以获取更多日志：
     • 修改 Dashboard 的启动命令，添加调试标志：
       JAVA_OPTS="-Drocketmq.console.debug=true"

[彭镜肇]

I have carefully reviewed your issue. It seems that the problem might be due to communication issues or state synchronization abnormalities between the Dashboard and the RocketMQ cluster. Here are my suggestions:

Possible Issues:

1. Consumer and producer states are not properly synchronized to the Broker

   • The consumer group might be offline because the consumers are not sending heartbeats to the Broker or the Dashboard cannot correctly access the Broker’s status data.
   • The producer group might not exist because the Dashboard is unable to retrieve the producer’s status information.

2. RocketMQ Dashboard and RocketMQ cluster communication issues

   • The Dashboard relies on the state information of the NameServer and the Broker. There might be network communication issues preventing the Dashboard from accessing the correct state.

3. RocketMQ cluster and Kubernetes network configuration issues

   • When deployed in Kubernetes, there might be external access or DNS configuration issues, causing communication failures between the Dashboard and the Broker. Typical problems include:
     • NameServer and Broker address resolution issues (e.g., internal IPs being returned).
     • Incorrect brokerIP1 or advertisedAddr configuration in the cluster.
     • Incorrect ROCKETMQ_NAMESRV_ADDR configuration in the Dashboard, which does not point to the correct NameServer.

4. The Broker does not support retrieving consumer/producer group states

   • Some RocketMQ versions rely on consumer and producer heartbeats to detect their states. By default, RocketMQ only records the state after heartbeats are sent.

Solutions:

1. Check and confirm your network configuration

   • Ensure that the RocketMQ deployment on Kubernetes is functioning properly.
   • Verify communication between the NameServer and the Broker:

     kubectl exec -it <broker-pod> -- curl http://<nameserver>:9876

   • Ensure that the ROCKETMQ_NAMESRV_ADDR configuration of the Dashboard is correct:
     • The RocketMQ NameServer address needs to be configured as an environment variable ROCKETMQ_NAMESRV_ADDR in the Dashboard container.
     • Check the configuration of the Dashboard:

       kubectl describe pod <dashboard-pod>

       Ensure that ROCKETMQ_NAMESRV_ADDR points to the correct NameServer address (e.g., rocketmq-nameserver.default.svc.cluster.local:9876).

2. Check the Broker configuration

   • Ensure that brokerIP1 and advertisedAddr in the Broker configuration are set correctly and point to the cluster's accessible address (not the container's private IP address).
     • In broker.conf, set:

       brokerIP1=<accessible IP>  
       brokerClusterName=DefaultCluster
       

     • If using Helm, you can configure brokerIP1 and advertisedAddr in the values.yaml file.
   • Check if the broker.conf configuration is complete, especially these key configurations:

     namesrvAddr=<NameServer address, e.g., rocketmq-nameserver.default.svc.cluster.local:9876>  
     listenPort=10911
     

   • Restart the Broker and observe whether the Broker and NameServer are successfully registered.

3. Check the status of consumers and producers

   • Consumer group status:
     • Consumer group status requires heartbeats to be sent to the Broker by the consumers. If consumers are not started properly or have not sent heartbeats for a long time, the consumer group will appear offline.
     • Ensure that the consumer application is running properly and consuming messages. Check the logs to see if the consumer is working correctly.
   • Producer status:
     • RocketMQ does not store static information about producer groups. The Broker only records a producer group after it sends messages.
     • Ensure that the producer application sends messages to the specified Topic and check the producer logs for any abnormalities.

4. Retrieve consumer and producer states manually

   • Use command-line tools to manually retrieve the state of consumers and producers to confirm if the issue lies with the Dashboard:
     • Retrieve the consumer group status:

       kubectl exec -it <broker-pod> -- bash
       sh mqadmin consumerStatus -n <nameserver address> -g <consumerGroup>

     • Retrieve the producer group status:

       sh mqadmin producerConnection -n <nameserver address> -g <producerGroup>

5. Check the compatibility of the Dashboard version

   • You are using Dashboard version 1.0.0, while the Broker is version 5.3.0. The Dashboard version is relatively old, which might cause compatibility issues.
   • Try upgrading the Dashboard to a version compatible with RocketMQ 5.3.0 (e.g., dashboard 1.1.0 or higher).

6. Check logs

   • NameServer logs: Check if the NameServer has successfully registered the Brokers and Consumers:

     kubectl logs <nameserver-pod>

   • Broker logs: Check if the Broker is receiving heartbeats from Consumers and Producers:

     kubectl logs <broker-pod>

   • Dashboard logs: Check if the Dashboard can access the NameServer and Brokers:

     kubectl logs <dashboard-pod>

7. Enable Debug Mode on the Dashboard

   • If the issue persists, you can try enabling debug mode on the RocketMQ Dashboard to get more detailed logs:
     • Modify the startup command of the Dashboard to add the debug flag:

       JAVA_OPTS="-Drocketmq.console.debug=true"
       

[siyuan]

我也是这个问题，解决了么，生产者都是匿名的，它那个生产者报错生产者组不存在可能是rocketmq5之前的版本报错，消费者组终端的报错就不清楚了

[Fish]

我也是这个问题，生产者给 topic 提交消息，然后消息的异常就是消费者不在线，但我的消费者已经启动了