Giters

mardambey / mypipe

MySQL binary log consumer with the ability to act on changed rows and publish changes to different systems with emphasis on Apache Kafka.

Home Page: http://mardambey.github.io/mypipe

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Subscription to multi table problem(specific)

ajinoline opened this issue · comments

commented

this's my application.overrides.conf

mypipe {

include-event-condition = """ (db == "medusa" && table =="forum_user") || (db == "medusa" && table.startsWith("forum_post_p"))"""

Avro schema repository client class name

schema-repo-client = "mypipe.avro.schema.SchemaRepo"

consumers represent sources for mysql binary logs

consumers {
database1 {
# database "host:port:user:pass" array
source = ["192.168.32.102:3309:mypipe:mypipe123"]
}
database2 {
#database "host:port:user:pass" array
source = ["192.168.32.102:3309:mypipe:mypipe123"]
}
}

data producers export data out (stdout, other stores, external services, etc.)

producers {
stdout {
class = "mypipe.producer.stdout.StdoutProducer"
}

kafka-generic {
  class = "mypipe.producer.KafkaMutationGenericAvroProducer"
}
kafka-specific-post-producer {
  class = "mypipe.producer.KafkaMutationSpecificAvroProducer"
}

kafka-specific-user-producer {
  class = "mypipe.producer.KafkaMutationSpecificAvroProducer"
}

}

pipes join consumers and producers

pipes {
kafka-specific-post {
enabled = true
consumers = ["database1"]
producer {
kafka-specific-post-producer {
schema-repo-client = "mypipe.avro.ForumPostSchemaRepo"
metadata-brokers = "192.168.32.29:9092"
}
}

  binlog-position-repo {
    class = "mypipe.api.repo.ConfigurableZookeeperBasedBinaryLogPositionRepository"
    config {
      path-prefix = "/mypipe-specific-post"
      conn = "192.168.32.29:2181,192.168.32.29:2182,192.168.32.29:2183"
      max-retry-seconds = 10
      data-dir = "forum-post"
    }
  }
}

}

pipes {
kafka-specific-user {
enabled = true
consumers = #["database2"]
producer {
kafka-specific-user-producer {
schema-repo-client = "mypipe.avro.ForumUsersSchemaRepo"
metadata-brokers = "192.168.32.29:9092"
}
}
binlog-position-repo {
class = "mypipe.api.repo.ConfigurableZookeeperBasedBinaryLogPositionRepository"
config {
path-prefix = "/mypipe-specific-user"
conn = "192.168.32.29:2181,192.168.32.29:2182,192.168.32.29:2183"
max-retry-seconds = 10
data-dir = "forum-user"
}
}
}
}

zookeeper used for Leader Election

zk {
leader-path = "/mypipe-leader-test"
conn = "192.168.32.29:2181,192.168.32.29:2182,192.168.32.29:2183"
max-retry-seconds = 10
}
}

object ForumUsersSchemaRepo extends InMemorySchemaRepo[Short, Schema] with ShortSchemaId with AvroSchema {

val DATABASE = "medusa"

val insert = new ForumUserInsert()
val update = new ForumUserUpdate()
val delete = new ForumUserDelete()

registerSchema(AvroSchemaUtils.specificSubject(DATABASE, s"forum_user", Mutation.InsertString), insert.getSchema)
registerSchema(AvroSchemaUtils.specificSubject(DATABASE, s"forum_user", Mutation.UpdateString), update.getSchema)
registerSchema(AvroSchemaUtils.specificSubject(DATABASE, s"forum_user", Mutation.DeleteString), delete.getSchema)

}
Execute update. I have two questions

One: logerror

this's Error message:

17:01:00.359 [ForkJoinPool-5-worker-13] ERROR m.p.KafkaMutationSpecificAvroProducer - failed to queue: None.get
scala.None$.get(Option.scala:347)
scala.None$.get(Option.scala:345)
mypipe.producer.KafkaMutationAvroProducer.queue(KafkaMutationAvroProducer.scala:92)
mypipe.producer.KafkaMutationAvroProducer$$anonfun$queueList$1.apply(KafkaMutationAvroProducer.scala:85)
mypipe.producer.KafkaMutationAvroProducer$$anonfun$queueList$1.apply(KafkaMutationAvroProducer.scala:85)
scala.collection.immutable.List.loop$3(List.scala:360)
scala.collection.immutable.List.dropWhile(List.scala:363)
mypipe.producer.KafkaMutationAvroProducer.queueList(KafkaMutationAvroProducer.scala:85)
mypipe.pipe.Pipe$$anon$5.onMutation(Pipe.scala:69)
mypipe.api.consumer.AbstractBinaryLogConsumer$$anonfun$4.apply(AbstractBinaryLogConsumer.scala:165)
mypipe.api.consumer.AbstractBinaryLogConsumer$$anonfun$4.apply(AbstractBinaryLogConsumer.scala:165)
mypipe.api.consumer.AbstractBinaryLogConsumer$$anonfun$processList$1.apply(AbstractBinaryLogConsumer.scala:193)
mypipe.api.consumer.AbstractBinaryLogConsumer$$anonfun$processList$1.apply(AbstractBinaryLogConsumer.scala:192)
scala.collection.LinearSeqOptimized$class.forall(LinearSeqOptimized.scala:83)
scala.collection.immutable.List.forall(List.scala:84)
mypipe.api.consumer.AbstractBinaryLogConsumer.processList(AbstractBinaryLogConsumer.scala:192)
mypipe.api.consumer.AbstractBinaryLogConsumer.commit(AbstractBinaryLogConsumer.scala:163)
mypipe.api.consumer.AbstractBinaryLogConsumer.handleXid(AbstractBinaryLogConsumer.scala:147)
mypipe.api.consumer.AbstractBinaryLogConsumer.handleEvent(AbstractBinaryLogConsumer.scala:36)
mypipe.mysql.AbstractMySQLBinaryLogConsumer$$anonfun$onStart$1$$anon$1.onEvent(AbstractMySQLBinaryLogConsumer.scala:132)
com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:788)
com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:648)
com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:405)
mypipe.mysql.AbstractMySQLBinaryLogConsumer$$anonfun$onStart$1$$anonfun$apply$mcZ$sp$1$$anonfun$1.apply$mcV$sp(AbstractMySQLBinaryLogConsumer.scala:145)
mypipe.mysql.AbstractMySQLBinaryLogConsumer$$anonfun$onStart$1$$anonfun$apply$mcZ$sp$1$$anonfun$1.apply(AbstractMySQLBinaryLogConsumer.scala:145)
mypipe.mysql.AbstractMySQLBinaryLogConsumer$$anonfun$onStart$1$$anonfun$apply$mcZ$sp$1$$anonfun$1.apply(AbstractMySQLBinaryLogConsumer.scala:145)
scala.util.Try$.apply(Try.scala:192)
mypipe.mysql.AbstractMySQLBinaryLogConsumer$$anonfun$onStart$1$$anonfun$apply$mcZ$sp$1.apply$mcV$sp(AbstractMySQLBinaryLogConsumer.scala:145)
mypipe.mysql.AbstractMySQLBinaryLogConsumer$$anonfun$onStart$1$$anonfun$apply$mcZ$sp$1.apply(AbstractMySQLBinaryLogConsumer.scala:145)
mypipe.mysql.AbstractMySQLBinaryLogConsumer$$anonfun$onStart$1$$anonfun$apply$mcZ$sp$1.apply(AbstractMySQLBinaryLogConsumer.scala:145)
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

17:01:00.359 [ForkJoinPool-5-worker-13] ERROR mypipe.mysql.ConfigBasedErrorHandler - Could not handle 4 mutation(s) from listener mypipe.pipe.Pipe$$anon$5@a45b029
17:01:00.359 [ForkJoinPool-5-worker-13] ERROR mypipe.mysql.MySQLBinaryLogConsumer - Consumer database1 failed to process event Some(XidEvent(17263008)) from Event{header=EventHeaderV4{timestamp=1464771660000, eventType=XID, serverId=321184, headerLength=19, dataLength=8, nextPosition=46711329, flags=0}, data=XidEventData{xid=17263008}}

From the abnormal point of view, is caused by latestSchemaCache. I check that subject is not empty.
Just like this : subject--->medusa_forum_user_update

Two:
Send Kafka message multiple table interaction. Send Kafka message when the other table operation message will accumulate.

Just like this(kafka consumer)：

medusaforum_post_p5��ʊn'�漸x+�@��U�Я��Я�җ��җ��ЌЌ��颜U��颜U�ϻ�ϻ��颜U��颜U

medusaforum_post_p5��ʊn'�漸x+�@��U�Я��Я�җ��җ��ЌЌ��颜U��颜U�ϻ�ϻ��颜U��颜U

medusaforum_post_p5��ʊn'�漸x+�@��U�Я��Я�җ��җ��ЌЌ��颜U��颜U�ϻ�ϻ��颜U��颜U

I should do something?