keeleys/canal_kafuka_example

mysql8/zookeeper/kafka/canal1.1.4

安装步骤

参考 https://github.com/alibaba/canal/wiki/QuickStart 依配置启动mysql，zookeeper,kafka, 启动canal

配置mysql (vim /usr/local/etc/my.cnf)

# Default Homebrew MySQL server config
[mysqld]
# Only allow connections from localhost
bind-address = 0.0.0.0
mysqlx-bind-address = 0.0.0.0

# 开启 binlog
log-bin=mysql-bin
# 选择 ROW 模式
binlog-format=ROW 
# 配置 MySQL replaction 需要定义，不要和 canal 的 slaveId 重复
server_id=1

安装zookeeper和kafka

## 启动
zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties & kafka-server-start /usr/local/etc/kafka/server.properties

安装canal

docker和本地启动二选一

# docker 启动 (172.16.101.72是本机ip地址,docker访问本机的)
docker run --name canal-server \
-e canal.instance.master.address=172.16.101.72:3306 \
-e canal.instance.dbUsername=canal \
-e canal.instance.dbPassword=canal \
-e canal.instance.filter.query.dml=true \
-e canal.instance.filter.regex='canal_tsdb.t_role,canal_tsdb.t_user' \
-e canal.serverMode=kafka \
-e canal.mq.topic=canal_default_topic \
-e canal.mq.dynamicTopic='canal_tsdb\\..*' \
-e canal.mq.servers=172.16.101.72:9092 \
-p 11111:11111 \
-d canal/canal-server:v1.1.4

# 查看日志
docker exec -it canal-server bash
tail -f /home/admin/canal-server/logs/canal/canal.log
tail -f /home/admin/canal-server/logs/example/example.log
tail -f /home/admin/canal-server/logs/example/meta.log

# canal.instance.filter.regex的书写格式
mysql 数据解析关注的表，Perl正则表达式.
多个正则之间以逗号(,)分隔，转义符需要双斜杠(\\) 
常见例子：
1.  所有表：.*   or  .*\\..*
2.  canal schema下所有表： canal\\..*
3.  canal下的以canal打头的表：canal\\.canal.*
4.  canal schema下的一张表：canal.test1
5.  多个规则组合使用：canal\\..*,mysql.test1,mysql.test2 (逗号分隔)

本地启动

# 下载 https://github.com/alibaba/canal/releases/download/canal-1.1.4/canal.deployer-1.1.4.tar.gz

# 参考上面的docker配吹，修改conf/example/instance.properties和conf/canal.properties
# 本地启动 进入canal目录

./bin/startup.sh

tail -f logs/canal/canal.log
tail -f logs/example/example.log

./bin/stop.sh

启动项目监听，然后修改数据记录看效果

spring-boot:run

FAQ

关于mysql8的canal账户,如果出现caching_sha2_password Auth failed问题，

-- 执行下面的解决
ALTER USER 'canal'@'%' IDENTIFIED BY 'canal' PASSWORD EXPIRE NEVER;
ALTER USER 'canal'@'%' IDENTIFIED WITH mysql_native_password BY 'canal';
FLUSH PRIVILEGES;

mysql 链接问题

ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'xxxxxx';

集群用的HA模式，多个canal-service注册同一个zk,通过zookeeper控制，只有一个服务器会真正启动的。查看注册用ls /otter/canal/cluster

运行日志

2020-09-27 18:05:16.535  INFO 44222 --- [ntainer#3-0-C-1] c.s.kafka.example.mq.UserTopicListener   : message: {"data":[{"id":"2","account":"","user_name":"0","password":"","age":null,"create_time":"2020-09-27 18:05:16","update_time":"2020-09-27 18:05:16"}],"database":"canal_tsdb","es":1601201116000,"id":2,"isDdl":false,"mysqlType":{"id":"int unsigned","account":"varchar(50)","user_name":"int","password":"varchar(50)","age":"INT","create_time":"timestamp","update_time":"timestamp"},"old":null,"pkNames":["id"],"sql":"","sqlType":{"id":4,"account":12,"user_name":4,"password":12,"age":4,"create_time":93,"update_time":93},"table":"t_user","ts":1601201116505,"type":"INSERT"}
2020-09-27 18:05:16.578  INFO 44222 --- [ntainer#3-0-C-1] c.s.kafka.example.mq.UserTopicListener   : table: t_user
2020-09-27 18:05:16.578  INFO 44222 --- [ntainer#3-0-C-1] c.s.kafka.example.mq.UserTopicListener   : before: null
2020-09-27 18:05:16.578  INFO 44222 --- [ntainer#3-0-C-1] c.s.kafka.example.mq.UserTopicListener   : after: [{"password":"","update_time":"2020-09-27 18:05:16","create_time":"2020-09-27 18:05:16","user_name":"0","id":"2","account":""}]
2020-09-27 18:05:24.824  INFO 44222 --- [ntainer#3-0-C-1] c.s.kafka.example.mq.UserTopicListener   : message: {"data":[{"id":"4","account":"222","user_name":"0","password":"","age":null,"create_time":"2020-09-27 18:05:24","update_time":"2020-09-27 18:05:24"}],"database":"canal_tsdb","es":1601201124000,"id":3,"isDdl":false,"mysqlType":{"id":"int unsigned","account":"varchar(50)","user_name":"int","password":"varchar(50)","age":"INT","create_time":"timestamp","update_time":"timestamp"},"old":null,"pkNames":["id"],"sql":"","sqlType":{"id":4,"account":12,"user_name":4,"password":12,"age":4,"create_time":93,"update_time":93},"table":"t_user","ts":1601201124819,"type":"INSERT"}
2020-09-27 18:05:24.825  INFO 44222 --- [ntainer#3-0-C-1] c.s.kafka.example.mq.UserTopicListener   : table: t_user
2020-09-27 18:05:24.825  INFO 44222 --- [ntainer#3-0-C-1] c.s.kafka.example.mq.UserTopicListener   : before: null
2020-09-27 18:05:24.825  INFO 44222 --- [ntainer#3-0-C-1] c.s.kafka.example.mq.UserTopicListener   : after: [{"password":"","update_time":"2020-09-27 18:05:24","create_time":"2020-09-27 18:05:24","user_name":"0","id":"4","account":"222"}]
2020-09-27 18:05:39.472  INFO 44222 --- [ntainer#3-0-C-1] c.s.kafka.example.mq.UserTopicListener   : message: {"data":[{"id":"2","account":"","user_name":"0","password":"","age":null,"create_time":"2020-09-27 18:05:16","update_time":"2020-09-27 18:05:16"}],"database":"canal_tsdb","es":1601201139000,"id":4,"isDdl":false,"mysqlType":{"id":"int unsigned","account":"varchar(50)","user_name":"int","password":"varchar(50)","age":"INT","create_time":"timestamp","update_time":"timestamp"},"old":null,"pkNames":["id"],"sql":"","sqlType":{"id":4,"account":12,"user_name":4,"password":12,"age":4,"create_time":93,"update_time":93},"table":"t_user","ts":1601201139467,"type":"DELETE"}
2020-09-27 18:05:39.472  INFO 44222 --- [ntainer#3-0-C-1] c.s.kafka.example.mq.UserTopicListener   : table: t_user
2020-09-27 18:05:39.473  INFO 44222 --- [ntainer#3-0-C-1] c.s.kafka.example.mq.UserTopicListener   : before: null
2020-09-27 18:05:39.473  INFO 44222 --- [ntainer#3-0-C-1] c.s.kafka.example.mq.UserTopicListener   : after: [{"password":"","update_time":"2020-09-27 18:05:16","create_time":"2020-09-27 18:05:16","user_name":"0","id":"2","account":""}]
2020-09-27 18:13:21.357  INFO 44222 --- [ntainer#3-0-C-1] c.s.kafka.example.mq.UserTopicListener   : message: {"data":[{"id":"4","account":"222","user_name":"1","password":"","age":null,"create_time":"2020-09-27 18:05:24","update_time":"2020-09-27 18:13:21"}],"database":"canal_tsdb","es":1601201601000,"id":5,"isDdl":false,"mysqlType":{"id":"int unsigned","account":"varchar(50)","user_name":"int","password":"varchar(50)","age":"INT","create_time":"timestamp","update_time":"timestamp"},"old":[{"user_name":"0","update_time":"2020-09-27 18:05:24"}],"pkNames":["id"],"sql":"","sqlType":{"id":4,"account":12,"user_name":4,"password":12,"age":4,"create_time":93,"update_time":93},"table":"t_user","ts":1601201601352,"type":"UPDATE"}
2020-09-27 18:13:21.358  INFO 44222 --- [ntainer#3-0-C-1] c.s.kafka.example.mq.UserTopicListener   : table: t_user
2020-09-27 18:13:21.358  INFO 44222 --- [ntainer#3-0-C-1] c.s.kafka.example.mq.UserTopicListener   : before: [{"update_time":"2020-09-27 18:05:24","user_name":"0"}]
2020-09-27 18:13:21.358  INFO 44222 --- [ntainer#3-0-C-1] c.s.kafka.example.mq.UserTopicListener   : after: [{"password":"","update_time":"2020-09-27 18:13:21","create_time":"2020-09-27 18:05:24","user_name":"1","id":"4","account":"222"}]

keeleys / canal_kafuka_example

mysql8/zookeeper/kafka/canal1.1.4

安装步骤

配置mysql (vim /usr/local/etc/my.cnf)

安装zookeeper和kafka

安装canal

启动项目监听，然后修改数据记录看效果

FAQ

运行日志

About

Languages