data-integration

There are 47 repositories under data-integration topic.

airflow
apache / airflow
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
airflow apache apache-airflow automation dag data-engineering data-integration data-orchestrator data-pipelines data-science elt etl machine-learning mlops orchestration python scheduler workflow workflow-engine workflow-orchestration
Language:Python 35973
airbytehq / airbyte
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
bigquery change-data-capture data data-analysis data-collection data-engineering data-integration data-pipeline elt etl java mssql mysql pipeline postgresql python redshift s3 self-hosted snowflake
Language:Python 15253
dagster-io / dagster
An orchestration platform for the development, production, and observation of data assets.
analytics dagster data-engineering data-integration data-orchestrator data-pipelines data-science etl metadata mlops orchestration python scheduler workflow workflow-automation
Language:Python 11008
Avaiga / taipy
Turns Data and AI algorithms into production-ready web applications in no time.
automation data-engineering data-integration data-ops data-visualization datascience developer-tools hacktoberfest hacktoberfest2023 job-scheduler mlops orchestration pipeline pipelines python scenario scenario-analysis taipy-core taipy-gui workflow
Language:Python 10864
apache / seatunnel
SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
apache batch cdc change-data-capture data-ingestion data-integration elt high-performance offline real-time streaming
Language:Java 7744
mage-ai / mage-ai
🧙 Build, run, and manage data pipelines for integrating and transforming data.
machine-learning artificial-intelligence data data-engineering data-science python elt etl pipelines data-pipelines orchestration data-integration sql spark dbt pipeline reverse-etl transformation
Language:Python 7621
kestra
kestra-io / kestra
Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.
workflow workflow-engine orchestration scheduler data-pipeline elt etl data data-engineering data-orchestration data-orchestrator low-code data-integration reverse-etl data-quality pipeline
Language:Java 7448
cloudquery
cloudquery / cloudquery
The open source high performance ELT framework powered by Apache Arrow
airbyte attack-surface-management aws azure bigquery cspm data data-analysis data-collection data-engineering data-integration elt etl etl-framework gcp github-api go google kubernetes sql
Language:Go 5758
apache / flink-cdc
Flink CDC is a streaming data integration tool
change-data-capture cdc batch data-integration data-pipeline distributed elt etl flink kafka mysql paimon postgresql real-time schema-evolution
Language:Java 5569
apache / hudi
Upserts, Deletes And Incremental Processing on Big Data.
hudi apachehudi datalake bigdata apachespark incremental-processing stream-processing data-integration apacheflink
Language:Java 5303
rudder-server
rudderlabs / rudder-server
Privacy and Security focused Segment-alternative, in Golang and React
privacy warehouse-management data-warehouse customer-data customer-data-pipeline customer-data-platform customer-data-lake segment-alternative data-integration data-synchronization etl bigquery redshift snowflake data-pipeline elt data-engineering cdp warehouse-native event-streaming
Language:Go 4038
jitsu
jitsucom / jitsu
Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days
data-integration clickhouse golang bigquery data-collection data-connectors redshift snowflake postgres
Language:TypeScript 3998
DTStack / chunjun
A data integration framework
bigdata data-integration flink framework java
Language:Java 3969
seandavi / awesome-single-cell
Community-curated list of software packages and data resources for single-cell, including RNA-seq, ATAC-seq, etc.
analysis analysis-pipeline atac-seq awesome-list bioinformatics cell-clusters cell-cycle cell-differentiation cell-populations clustering data-integration data-visualization dimensionality-reduction gene-expression gene-expression-profiles python rna-seq-data rna-seq-experiments scrna-seq-data single-cell
3091
fluvio
infinyon / fluvio
Lean and mean distributed stream processing system written in rust and web assembly. Alternative to Kafka + Flink in one.
cloud-native data-flow data-integration data-pipelines distributed-systems event-driven-architecture real-time rust serverless stateful stream-processing stream-processing-engine streaming streaming-data streaming-data-pipelines streaming-data-processing webassembly
Language:Rust 2871
apache / incubator-devlake
Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
dashboard-friendly data data-analysis data-engineering data-integration data-transfers devops domain-layer dora etl golang hacktoberfest integration jira open-source user-friendly
Language:Go 2544
bruin-data / ingestr
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
bigquery copy-database data-ingestion data-integration data-pipeline duckdb ingestion-pipeline mssql postgresql snowflake
Language:Python 2447
mara / mara-pipelines
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
data data-integration etl pipeline postgresql python
Language:Python 2067
bytedance / bitsail
BitSail is a distributed high-performance data integration engine which supports batch, streaming and incremental scenarios. BitSail is widely used to synchronize hundreds of trillions of data every day.
flink big-data data-integration data-lake data-pipeline data-synchronization high-performance real-time
Language:Java 1608
apache / hop
Hop Orchestration Platform
java streaming hop apache data-integration etl orchestration pipeline workflow
Language:Java 915
kuwala
kuwala-io / kuwala
Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data science models and products with a focus on geospatial data. Currently, the following data connectors are available worldwide: a) High-resolution demographics data b) Point of Interests from Open Street Map c) Google Popular Times
data data-integration data-science open-data spatial-analysis elt kuwala open-source scraping dbt postgres pyspark python jupyter admin-boundaries google-trends population react no-code react-flow
Language:JavaScript 783
artie-labs / transfer
Database replication platform that leverages change data capture. Stream production data from databases to your data warehouse (Snowflake, BigQuery, Redshift) in real-time.
snowflake cdc change-data-capture golang bigquery kafka apache-kafka data-integration data-pipelines database debezium elt redshift
Language:Go 579
leesf / hudi-resources
汇总Apache Hudi相关资料
hudi apachehudi apache hudi-resources datalake bigdata stream-processing incremental-processing data-integration
533
immunogenomics / harmony
Fast, sensitive and accurate integration of single-cell data with Harmony
scrna-seq algorithm data-integration r
Language:R 510
apache / seatunnel-web
SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).
apache data-integration data-pipeline etl-framework high-performance offline real-time seatunnel sql-engine
Language:Java 491
saeyslab / nichenetr
NicheNet: predict active ligand-target links between interacting cells
single-cell-omics intercellular-communication ligand-receptor ligand-target data-integration single-cell-rna-seq rna-seq gene-expression network-inference cell-cell-communication
Language:R 466
conduit
ConduitIO / conduit
Conduit streams data between data stores. Kafka Connect replacement. No JVM required.
go data-integration etl data-pipeline data-engineering data-stream conduit kafka kafkaconnect
Language:Go 374
theislab / scarches
Reference mapping for single-cell genomics
single-cell deep-learning rna-seq-analysis data-integration batch-correction multimodal-deep-learning multiomics single-cell-genomics scrna-seq human-cell-atlas
Language:Jupyter Notebook 328
recap
gabledata / recap
Work with your web service, database, and streaming schemas in a single format.
data-catalog metadata data-discovery recap data-engineering data-integration data-pipelines etl
Language:Python 322
CategoricalData / CQL
Categorical Query Language IDE
cql category-theory functional-programming data-migratation etl data-integration data-science data-exchange
Language:Java 297
swimos / swim-rust
Self-contained distributed software platform for building stateful, massively real-time streaming applications in Rust.
actor-model async decentralized-applications distributed-systems real-time rust stateful web data-integration framework kafka stream-processing streaming streaming-data-pipelines serverless
Language:Rust 286
cuebook / cuelake
Use SQL to build ELT pipelines on a data lakehouse.
apache-iceberg delta lakehouse datalake data-lake elt etl data-engineering data-integration data-ingestion apache-spark spark-sql upsert incremental-updates data-transfer pipelines data-pipeline zeppelin-notebook sql
Language:JavaScript 283
hetio / hetionet
Hetionet: an integrative network of disease
hetionet rephetio network neo4j hetnet drug-repurposing data-integration
Language:HTML 253
dataplane-app / dataplane
Dataplane is an Airflow inspired unified data platform with additional data mesh and RPA capability to automate, schedule and design data pipelines and workflows. Dataplane is written in Golang with a React front end.
golang airflow kubernetes data scheduler workflows workflow-automation data-pipelines data-science data-engineering workflow data-analysis data-integration dataplane datawarehouse etl pipelines robotics-process-automation rpa finance
Language:JavaScript 209
opensanctions / nomenklatura
Framework and command-line tools for integrating FollowTheMoney data streams from multiple sources
data-integration deduplication record-link
Language:Python 194
slowkow / harmonypy
🎼 Integrate multiple high-dimensional datasets with fuzzy k-means and locally linear adjustments.
bioinformatics data-integration data-science single-cell-analysis
Language:Python 189