acheng-floyd / mysql2ch

Sync data from MySQL to ClickHouse.

mysql2ch

中文文档

Introduction

mysql2ch to sync data from MySQL to ClickHouse, support full and increment.

Features

• Full data etl and continuous sync.
• Support DDL and DML sync,current support add column and drop column of DDL, and full support of DML also.
• Rich configurable items.
• Consumer and producer monitor ui.

Requirements

• kafka,message queue to store mysql binlog event.
• redis,cache mysql binlog file and position and store monitor data.

Install

$ pip install mysql2ch

Usage

Make a .env file in execute dir or set system environment variable:

Create .env

Example .env.

Full data etl

Maybe you need make full data etl before continuous sync data from MySQL to ClickHouse or redo data etl with --renew.

$ mysql2ch etl -h

usage: mysql2ch etl [-h] --schema SCHEMA [--tables TABLES] [--renew]

optional arguments:
  -h, --help       show this help message and exit
  --schema SCHEMA  Schema to full etl.
  --tables TABLES  Tables to full etl,multiple tables split with comma.
  --renew          Etl after try to drop the target tables.

Produce

Listen all MySQL binlog and produce to kafka.

$ mysql2ch produce

Consume

Consume message from kafka and insert to ClickHouse,and you can skip error with --skip-error.

$ mysql2ch consume -h

usage: mysql2ch consume [-h] --schema SCHEMA [--skip-error] [--auto-offset-reset AUTO_OFFSET_RESET]

optional arguments:
  -h, --help            show this help message and exit
  --schema SCHEMA       Schema to consume.
  --skip-error          Skip error rows.
  --auto-offset-reset AUTO_OFFSET_RESET
                        Kafka auto offset reset,default earliest.

Monitor UI

$ mysql2ch ui -h

usage: mysql2ch ui [-h] [--host HOST] [-p PORT]

optional arguments:
  -h, --help            show this help message and exit
  --host HOST           Listen host.
  -p PORT, --port PORT  Listen port.

Use docker-compose(recommended)

Example docker-compose.yml.

Optional

Sentry,error reporting,worked if set SENTRY_DSN in .env.

License

This project is licensed under the MIT License.