received duplicate messages, branch: master

Question

received duplicate messages, branch: master

fengxiao2019 opened this issue 3 years ago · comments

fengxiao2019 commented 3 years ago

the duplicated message id： 0A64F892048A1B9871B45B0C6972005C

2021-Dec-18 16:19:00.037053:processResponseCommand, code:11, opaque:1028, maxRetryTimes:1, retrySendTimes:1[processResponseCommand:520]
2021-Dec-18 16:19:00.037081:cancelTimerCallback: opaque:1028[cancelTimerCallback:625]
2021-Dec-18 16:19:00.037153:handleAsyncRequestTimeout aborted opaque:1028, e_code:125, msg:操作已取消[handleAsyncRequestTimeout:538]
2021-Dec-18 16:19:00.037315:incConsumeRT before Key:zhujiuyin_async_topic@zhujiuyin_async_consumer, RT:13608, Count: 1[incConsumeRT:103]
2021-Dec-18 16:19:00.037352:incConsumeRT After Key:zhujiuyin_async_topic@zhujiuyin_async_consumer, RT:13608, Count: 1[incConsumeRT:113]
2021-Dec-18 16:19:00.037385:PullRequest: putMessage m_queueOffsetMax:3025 [putMessage:66]
2021-Dec-18 16:19:00.037470:FOUND:MessageQueue [topic=zhujiuyin_async_topic, brokerName=RaftNode04, queueId=4] with size:1, nextBeginOffset:3026[onSuccess:88]
2021-Dec-18 16:19:00.037159:code:0, language:JAVA, version:393, opaque:1027, flag:1, remark:FOUND, headLen:219, bodyLen:338 [Decode:177]
2021-Dec-18 16:19:00.037514:=====Receive Messages,Topic[zhujiuyin_async_topic], MsgId[0A64F892048A1B9871B45B0C6972005C],Body[{"app_name": "server", "script_name": "crontab_check_rocketmq_lag", "param": null, "log_tag": null}],RetryTimes[0][ConsumeRequest:179]
2021-Dec-18 16:19:00.037572:find_response opaque:1027[ProcessData:507]
2021-Dec-18 16:19:00.037610:processResponseCommand, code:11, opaque:1027, maxRetryTimes:1, retrySendTimes:1[processResponseCommand:520]
2021-Dec-18 16:19:00.037631:cancelTimerCallback: opaque:1027[cancelTimerCallback:625]
2021-Dec-18 16:19:00.037685:code:0, language:JAVA, version:393, opaque:1029, flag:1, remark:FOUND, headLen:219, bodyLen:338 [Decode:177]
2021-Dec-18 16:19:00.037717:find_response opaque:1029[ProcessData:507]
2021-Dec-18 16:19:00.037739:processResponseCommand, code:11, opaque:1029, maxRetryTimes:1, retrySendTimes:1[processResponseCommand:520]
2021-Dec-18 16:19:00.037757:cancelTimerCallback: opaque:1029[cancelTimerCallback:625]
2021-Dec-18 16:19:00.037862:incConsumeRT before Key:zhujiuyin_async_topic@zhujiuyin_async_consumer, RT:13608, Count: 1[incConsumeRT:103]
2021-Dec-18 16:19:00.037897:incConsumeRT After Key:zhujiuyin_async_topic@zhujiuyin_async_consumer, RT:27216, Count: 2[incConsumeRT:113]
2021-Dec-18 16:19:00.037931:PullRequest: putMessage m_queueOffsetMax:3025 [putMessage:66]
2021-Dec-18 16:19:00.037984:FOUND:MessageQueue [topic=zhujiuyin_async_topic, brokerName=RaftNode04, queueId=4] with size:1, nextBeginOffset:3026[onSuccess:88]
2021-Dec-18 16:19:00.038067:incConsumeRT before Key:zhujiuyin_async_topic@zhujiuyin_async_consumer, RT:13609, Count: 1[incConsumeRT:103]
2021-Dec-18 16:19:00.038102:incConsumeRT After Key:zhujiuyin_async_topic@zhujiuyin_async_consumer, RT:40825, Count: 3[incConsumeRT:113]
2021-Dec-18 16:19:00.038126:PullRequest: putMessage m_queueOffsetMax:3025 [putMessage:66]
2021-Dec-18 16:19:00.038163:FOUND:MessageQueue [topic=zhujiuyin_async_topic, brokerName=RaftNode04, queueId=4] with size:1, nextBeginOffset:3026[onSuccess:88]
2021-Dec-18 16:19:00.038242:handleAsyncRequestTimeout aborted opaque:1027, e_code:125, msg:操作已取消[handleAsyncRequestTimeout:538]
2021-Dec-18 16:19:00.038274:handleAsyncRequestTimeout aborted opaque:1029, e_code:125, msg:操作已取消[handleAsyncRequestTimeout:538]
2021-Dec-18 16:19:00.038413:before insert declared filed,MAP SIZE is:2[doBeforeRequest:43]
2021-Dec-18 16:19:00.038454:after insert declared filed, MAP SIZE is:12[doBeforeRequest:47]
2021-Dec-18 16:19:00.038473:total msg info are:qadeptRMQ3025zhujiuyin_async_consumer32430261639814925133150007zhujiuyin_async_topic, size is:75[doBeforeRequest:63]
2021-Dec-18 16:19:00.038643:invokeAsync success, addr:xx.xx.xx.xx:50911, code:11, opaque:1037[invokeAsync:265]
2021-Dec-18 16:19:00.038715:before insert declared filed,MAP SIZE is:2[doBeforeRequest:43]
2021-Dec-18 16:19:00.038748:after insert declared filed, MAP SIZE is:12[doBeforeRequest:47]
2021-Dec-18 16:19:00.038768:total msg info are:qadeptRMQ3025zhujiuyin_async_consumer32430261639814925133150007zhujiuyin_async_topic, size is:75[doBeforeRequest:63]
2021-Dec-18 16:19:00.038870:invokeAsync success, addr:xx.xx.xx.xx:50911, code:11, opaque:1038[invokeAsync:265]
2021-Dec-18 16:19:00.038934:before insert declared filed,MAP SIZE is:2[doBeforeRequest:43]
2021-Dec-18 16:19:00.038978:after insert declared filed, MAP SIZE is:12[doBeforeRequest:47]
2021-Dec-18 16:19:00.039006:total msg info are:qadeptRMQ3025zhujiuyin_async_consumer32430261639814925133*150007zhujiuyin_async_topic, size is:75[doBeforeRequest:63]
2021-Dec-18 16:19:00.039151:invokeAsync success, addr:xx.xx.xx.xx:50911, code:11, opaque:1039[invokeAsync:265]
2021-Dec-18 16:19:00.079686:incConsumeRT before Key:zhujiuyin_async_topic@zhujiuyin_async_consumer, RT:42, Count: 1[incConsumeRT:103]
2021-Dec-18 16:19:00.079877:incConsumeRT After Key:zhujiuyin_async_topic@zhujiuyin_async_consumer, RT:40867, Count: 4[incConsumeRT:113]
2021-Dec-18 16:19:00.079912:incConsumeOKTPS Before Key:zhujiuyin_async_topic@zhujiuyin_async_consumer, Count: 1[incConsumeOKTPS:117]
2021-Dec-18 16:19:00.079931:incConsumeOKTPS After Key:zhujiuyin_async_topic@zhujiuyin_async_consumer, Count: 1[incConsumeOKTPS:124]
2021-Dec-18 16:19:00.079950:m_queueOffsetMax is:3025[removeMessage:111]
2021-Dec-18 16:19:00.079979: offset result is:3026, m_queueOffsetMax is:3025, msgs size:1[removeMessage:115]
2021-Dec-18 16:19:00.079998:remove these msg from m_msgTreeMap, its offset:3025[removeMessage:118]
2021-Dec-18 16:19:00.080093:=====Receive Messages,Topic[zhujiuyin_async_topic], MsgId[0A64F892048A1B9871B45B0C6972005C],Body[{"app_name": "server", "script_name": "crontab_check_rocketmq_lag", "param": null, "log_tag": null}],RetryTimes[0][ConsumeRequest:179]
2021-Dec-18 16:19:00.082514:incConsumeRT before Key:zhujiuyin_async_topic@zhujiuyin_async_consumer, RT:2, Count: 1[incConsumeRT:103]
2021-Dec-18 16:19:00.082592:incConsumeRT After Key:zhujiuyin_async_topic@zhujiuyin_async_consumer, RT:40869, Count: 5[incConsumeRT:113]
2021-Dec-18 16:19:00.082613:incConsumeOKTPS Before Key:zhujiuyin_async_topic@zhujiuyin_async_consumer, Count: 1[incConsumeOKTPS:117]
2021-Dec-18 16:19:00.082630:incConsumeOKTPS After Key:zhujiuyin_async_topic@zhujiuyin_async_consumer, Count: 2[incConsumeOKTPS:124]
2021-Dec-18 16:19:00.082647:m_queueOffsetMax is:3025[removeMessage:111]
2021-Dec-18 16:19:00.082668:Note: Get local offset for mq:MessageQueue [topic=zhujiuyin_async_topic, brokerName=RaftNode04, queueId=4] failed, may be it is updated before. skip..[ConsumeRequest:316]
2021-Dec-18 16:19:00.082717:=====Receive Messages,Topic[zhujiuyin_async_topic], MsgId[0A64F892048A1B9871B45B0C6972005C],Body[{"app_name": "server", "script_name": "crontab_check_rocketmq_lag", "param": null, "log_tag": null}],RetryTimes[0][ConsumeRequest:179]
2021-Dec-18 16:19:00.084563:incConsumeRT before Key:zhujiuyin_async_topic@zhujiuyin_async_consumer, RT:2, Count: 1[incConsumeRT:103]
2021-Dec-18 16:19:00.084614:incConsumeRT After Key:zhujiuyin_async_topic@zhujiuyin_async_consumer, RT:40871, Count: 6[incConsumeRT:113]
2021-Dec-18 16:19:00.084635:incConsumeOKTPS Before Key:zhujiuyin_async_topic@zhujiuyin_async_consumer, Count: 1[incConsumeOKTPS:117]
2021-Dec-18 16:19:00.084651:incConsumeOKTPS After Key:zhujiuyin_async_topic@zhujiuyin_async_consumer, Count: 3[incConsumeOKTPS:124]
2021-Dec-18 16:19:00.084668:m_queueOffsetMax is:3025[removeMessage:111]
2021-Dec-18 16:19:00.084686:Note: Get local offset for mq:MessageQueue [topic=zhujiuyin_async_topic, brokerName=RaftNode04, queueId=4] failed, may be it is updated before. skip..[ConsumeRequest:316]
2021-Dec-18 16:19:00.201338:Client factory start persistAllConsumerOffset[persistAllConsumerOffset:811]
2021-Dec-18 16:19:00.201573:oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=zhujiuyin_async_topic, brokerName=RaftNode04, queueId=4], its offset is:3026[updateConsumeOffsetToBroker:310]
2021-Dec-18 16:19:00.201633:before insert declared filed,MAP SIZE is:2[doBeforeRequest:43]
2021-Dec-18 16:19:00.201668:after insert declared filed, MAP SIZE is:6[doBeforeRequest:47]
2021-Dec-18 16:19:00.201694:total msg info are:qadeptRMQ3026zhujiuyin_async_consumer4zhujiuyin_async_topic, size is:49[doBeforeRequest:63]
2021-Dec-18 16:19:00.201919:invokeOneway success. addr:xx.xx.xx.xx:50911, code:15[invokeOneway:281]
2021-Dec-18 16:19:00.202328:had received all data. msgLen:117, from:45, recvLen:121[readNextMessageIntCallback:262]
2021-Dec-18 16:19:00.202429:too little data received with sum = 0[readNextMessageIntCallback:254]
2021-Dec-18 16:19:00.202484:code:0, language:JAVA, version:393, opaque:1040, flag:1, remark:, headLen:113, bodyLen:0 [Decode:177]
2021-Dec-18 16:19:00.202543:responseFuture was deleted by timeout of opaque:1040[ProcessData:502]
2021-Dec-18 16:19:03.164889:had received all data. msgLen:291, from:24, recvLen:295[readNextMessageIntCallback:262]
2021-Dec-18 16:19:03.165150:too little data received with sum = 0[readNextMessageIntCallback:254]
2021-Dec-18 16:19:03.165265:code:307, language:JAVA, version:393, opaque:-591779724, flag:0, remark:, headLen:287, bodyLen:0 [Decode:177]
2021-Dec-18 16:19:03.165430:request Command received:processRequest, addr::30911, code:307[processRequest:31]
2021-Dec-18 16:19:03.165469:consumerGroup:zhujiuyin_async_consumer, clientId:65.22@1406#DEFAULT, jstackEnable:0[Decode:624]
2021-Dec-18 16:19:03.165488:getConsumerRunningInfo:zhujiuyin_async_consumer[getConsumerRunningInfo:111]
2021-Dec-18 16:19:03.165627:getConsumeStats Topic:%RETRY%zhujiuyin_async_consumer, Group:zhujiuyin_async_consumer[getConsumeStats:74]
2021-Dec-18 16:19:03.165650:getConsumeStats Key:%RETRY%zhujiuyin_async_consumer@zhujiuyin_async_consumer[getConsumeStats:77]
2021-Dec-18 16:19:03.165677:getConsumeStats Topic:zhujiuyin_async_topic, Group:zhujiuyin_async_consumer[getConsumeStats:74]
2021-Dec-18 16:19:03.165693:getConsumeStats Key:zhujiuyin_async_topic@zhujiuyin_async_consumer[getConsumeStats:77]
2021-Dec-18 16:19:03.165899:before insert declared filed,MAP SIZE is:2[doBeforeRequest:43]

James Yean · Answer 1 · Mon Dec 20 2021 12:48:09 GMT+0800 (China Standard Time)

@fengxiao2019 Could you describe your case in simple sentences?

fengxiao2019 · Answer 2 · Mon Dec 20 2021 12:54:22 GMT+0800 (China Standard Time)

Multiple threads with the same process will receive messages with the same message_id at the same time (or at many intervals, usually at the millisecond level).
Single process with a single thread: it will receive the same message_id several times in a row after the current message_id has been processed.
This scenario is still quite frequent.

同一个进程多个线程，会在同一时刻（或者间隔很多的时间内，通常是在毫秒级别）收到相同message_id的消息。
单个进程单个线程：会在当前message_id处理完之后，连续收到多次同一个message_id的消息。
这种场景还是挺频繁的。

fengxiao2019 · Answer 3 · Mon Dec 20 2021 12:55:51 GMT+0800 (China Standard Time)

I compiled the version with the master branch

James Yean · Answer 4 · Mon Dec 20 2021 13:01:42 GMT+0800 (China Standard Time)

@fengxiao2019 Do you operate consumer cluster before receive duplicated messages?

fengxiao2019 · Answer 5 · Mon Dec 20 2021 13:06:11 GMT+0800 (China Standard Time)

no, I didn't.

chwlsh · Answer 6 · Sat Oct 08 2022 11:44:33 GMT+0800 (China Standard Time)

遇到了同样的问题，64个写队列和64个读队列的一个消费者组设置10个进程每个进程10个线程同一个进程会在同一时间重复消费同一个消息

2022-10-08 11:42:59.831 [PID:89365] root log.py[line:210] INFO [index-89365] [msg-id-0AFF001500010000000026901db87443] begin process 
2022-10-08 11:42:59.831 [PID:89365] root log.py[line:210] INFO [index-89365] [msg-id-0AFF001500010000000026901db87443] begin process 
2022-10-08 11:42:59.831 [PID:89365] root log.py[line:210] INFO [index-89365] [msg-id-0AFF001500010000000026901db87443] begin process 
2022-10-08 11:42:59.831 [PID:89365] root log.py[line:210] INFO [index-89365] [msg-id-0AFF001500010000000026901db87443] begin process

beibei198918 · Answer 7 · Tue Feb 28 2023 10:58:45 GMT+0800 (China Standard Time)

这个重复消费的问题是commit 5629226 引起的，但我不知道怎么解决解决

StruggleLinzi · Answer 8 · Mon Jul 03 2023 16:57:38 GMT+0800 (China Standard Time)

这个重复消费的问题是commit 5629226 引起的，但我不知道怎么解决解决

你好，请问这个问题解决了吗？