[URGENT] [Performance] uReplicator's very slow & have exception "Doesn't match ExternalView" when use enableAutoWhitelist TRUE
dungnt081191 opened this issue · comments
dungnt081191 commented
Hi everybody ,
I met this ERROR on CONTROLLER when run uReplicator with this condition:
- Kubernetes ( 2 pod Controller + 3 pod Worker)
- enableAutoWhitelist TRUE
- Handle around ~200 topic from Source Kafka to Destination Kafka
Actual result :
- uReplicator's very slow
- message in Destination topic not sync in real time from Source topic ( ~45 minutes )
- have exception "Doesn't match ExternalView" when use enableAutoWhitelist TRUE
- Worker restart lots of time ( ~80 times in a half of day )
Expected result :
- message will be replicated in real time
- no exception about "Doesn't match ExternalView"
Any idea / How to fix this ? @yangy0000 @Technoboy- @yurishkuro
##Log of Controller :
[2019-12-16 09:08:03,299] INFO Unregistering ClusterStatus: cluster=uReplicator-cluster-name,instanceName=HelixMirrorMaker-1576486040595,resourceName=topic-replicate-xxx.qc (org.apache.helix.monitoring.mbeans.ClusterStatusMonitor:186)
[2019-12-16 09:08:03,302] INFO Unregistering ClusterStatus: cluster=uReplicator-cluster-name,instanceName=HelixMirrorMaker-1576486040595,resourceName=topic-replicate-xxx (org.apache.helix.monitoring.mbeans.ClusterStatusMonitor:186)
[2019-12-16 09:08:33,193] ERROR For topic:topic-replicate-xxx, number of partitions for IdealState: topic-replicate-xxx, {IDEAL_STATE_MODE=CUSTOMIZED, MAX_PARTITIONS_PER_INSTANCE=1, NUM_PARTITIONS=1, REBALANCE_MODE=CUSTOMIZED, REPLICAS=1, STATE_MODEL_DEF_REF=OnlineOffline, STATE_MODEL_FACTORY_NAME=DEFAULT}{0={HelixMirrorMaker-1576486040595=ONLINE}}{} doesn't match ExternalView: topic-replicate-xxx, {BUCKET_SIZE=0, IDEAL_STATE_MODE=CUSTOMIZED, MAX_PARTITIONS_PER_INSTANCE=1, NUM_PARTITIONS=1, REBALANCE_MODE=CUSTOMIZED, REPLICAS=1, STATE_MODEL_DEF_REF=OnlineOffline, STATE_MODEL_FACTORY_NAME=DEFAULT}{}{} (com.uber.stream.kafka.mirrormaker.controller.validation.ValidationManager:177)
Jiwei Guo commented
Worker ip is changing or not after restart?
dungnt081191 commented
Hi @Technoboy-
yes, Worker IP is change everytime after restart.
What's happen if Worker IP is always change like that .
I saw Controller log have a long list connected worker ( maybe include ex-worker not available anymore ) . How can i prevent it ?