FlinkX

English | 中文

Communication

We are recruiting Big data platform development engineers.If you want more information about the position, please add WeChat ID [ysqwhiletrue] or email your resume to sishu@dtstack.com.
We use DingTalk to communicate,You can search the group number [30537511] or scan the QR code below to join the communication group

Introduction

FlinkX is a data synchronization tool based on Flink. FlinkX can collect static data, such as MySQL, HDFS, etc, as well as real-time changing data, such as MySQL binlog, Kafka, etc. At the same time, FlinkX is also a computing framework that supports all the syntax and features of native FlinkSql , And provide a large number of cases. FlinkX currently includes the following features:

Most plugins support concurrent reading and writing of data, which can greatly improve the speed of reading and writing;
Some plug-ins support the function of failure recovery, which can restore tasks from the failed location and save running time; Failure Recovery
The source plugin for relational databases supports interval polling. It can continuously collect changing data; Interval Polling
Some databases support opening Kerberos security authentication; Kerberos
Limit the reading speed of source plugins and reduce the impact on business databases;
Save the dirty data when writing data;
Limit the maximum number of dirty data;
Multiple running modes: Local,Standalone,Yarn Session,Yarn Per;
Synchronization tasks support transformer operations that execute flinksql syntax;
sql task support is shared with flinkSql's own connectors;

The following databases are currently supported:

	Database Type	Source	Sink	Lookup
Batch Synchronization	MySQL	doc	doc	doc
	TiDB		reference mysql	reference mysql
	Oracle	doc	doc	doc
	Doris		doc
	SqlServer	doc	doc	doc
	PostgreSQL	doc	doc	doc
	DB2	doc	doc	doc
	ClickHouse	doc	doc	doc
	Greenplum	doc	doc
	KingBase	doc	doc
	MongoDB	doc	doc	doc
	SAP HANA	doc	doc
	ElasticSearch7	doc	doc	doc
	FTP	doc	doc
	HDFS	doc	doc
	Stream	doc	doc
	Redis		doc	doc
	Hive		doc
	Solr	doc	doc
	File	doc
Stream Synchronization	Kafka	doc	doc
	EMQX	doc	doc
	MySQL Binlog	doc
	Oracle LogMiner	doc
	Sqlserver CDC	doc
	Postgres CDC	doc