FlinkX
English | 中文
Communication
-
We are recruiting Big data platform development engineers.If you want more information about the position, please add WeChat ID [ysqwhiletrue] or email your resume to sishu@dtstack.com.
-
We use DingTalk to communicate,You can search the group number [30537511] or scan the QR code below to join the communication group
Introduction
FlinkX is a data synchronization tool based on Flink. FlinkX can collect static data, such as MySQL, HDFS, etc, as well as real-time changing data, such as MySQL binlog, Kafka, etc. FlinkX currently includes the following features:
-
Most plugins support concurrent reading and writing of data, which can greatly improve the speed of reading and writing;
-
Some plug-ins support the function of failure recovery, which can restore tasks from the failed location and save running time; Failure Recovery
-
The Reader plugin for relational databases supports interval polling. It can continuously collect changing data; Interval Polling
-
Some databases support opening Kerberos security authentication; Kerberos
-
Limit the reading speed of Reader plugins and reduce the impact on business databases;
-
Save the dirty data when writing data;
-
Limit the maximum number of dirty data;
-
Multiple running modes: Local,Standalone,Yarn Session,Yarn Per;
The following databases are currently supported:
Database Type | Reader | Writer | |
---|---|---|---|
Batch Synchronization | MySQL | doc | doc |
Oracle | doc | doc | |
SqlServer | doc | doc | |
PostgreSQL | doc | doc | |
DB2 | doc | doc | |
GBase | doc | doc | |
ClickHouse | doc | doc | |
PolarDB | doc | doc | |
SAP Hana | doc | doc | |
Teradata | doc | doc | |
Phoenix | doc | doc | |
达梦 | doc | doc | |
Cassandra | doc | doc | |
ODPS | doc | doc | |
HBase | doc | doc | |
MongoDB | doc | doc | |
Kudu | doc | doc | |
ElasticSearch | doc | doc | |
FTP | doc | doc | |
HDFS | doc | doc | |
Carbondata | doc | doc | |
Redis | doc | ||
Hive | doc | ||
Stream Synchronization | Kafka | doc | doc |
EMQX | doc | doc | |
MySQL Binlog | doc | ||
MongoDB Oplog | doc | ||
PostgreSQL WAL | doc |
Documentation
Documentation | Old Documentation
License
FlinkX is under the Apache 2.0 license. See the LICENSE file for details.