data-integration

There are 63 repositories under data-integration topic.

airflow
apache / airflow
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
airflow apache apache-airflow python scheduler workflow automation dag data-engineering data-integration data-orchestrator data-pipelines data-science elt etl machine-learning mlops orchestration workflow-engine workflow-orchestration
Language:Python 43122
airbytehq / airbyte
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
bigquery change-data-capture data data-analysis data-collection data-engineering data-integration data-pipeline elt etl java mssql mysql pipeline postgresql python redshift s3 self-hosted snowflake
Language:Python 19980
taipy
Avaiga / taipy
Turns Data and AI algorithms into production-ready web applications in no time.
automation data-engineering data-integration data-ops data-visualization datascience developer-tools hacktoberfest hacktoberfest2023 job-scheduler mlops orchestration pipeline pipelines python scenario scenario-analysis taipy-core taipy-gui workflow
Language:Python 18849
dagster-io / dagster
An orchestration platform for the development, production, and observation of data assets.
analytics dagster data-engineering data-integration data-orchestrator data-pipelines data-science etl hacktoberfest metadata mlops orchestration python scheduler workflow workflow-automation
Language:Python 14365
apache / seatunnel
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
data-integration high-performance offline real-time apache batch cdc change-data-capture data-ingestion elt streaming embeddings llm multimodal
Language:Java 8867
mage-ai / mage-ai
🧙 Build, run, and manage data pipelines for integrating and transforming data.
artificial-intelligence data data-engineering data-integration data-pipelines data-science dbt elt etl machine-learning orchestration pipeline pipelines python reverse-etl spark sql transformation
Language:Python 8517
apache / flink-cdc
Flink CDC is a streaming data integration tool
change-data-capture cdc batch data-integration data-pipeline distributed elt etl flink kafka mysql paimon postgresql real-time schema-evolution
Language:Java 6259
cloudquery
cloudquery / cloudquery
Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
aws gcp azure sql data-integration elt etl etl-framework bigquery data-collection data-engineering kubernetes data airbyte github-api data-analysis google go cspm attack-surface-management
Language:Go 6249
apache / hudi
Upserts, Deletes And Incremental Processing on Big Data.
apacheflink apachehudi apachespark bigdata data-integration datalake hudi incremental-processing stream-processing
Language:Java 6011
fluvio
infinyon / fluvio
🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.
cloud-native data-analytics data-flow data-integration data-pipelines distributed-systems event-driven-architecture real-time rust serverless stateful stream-processing stream-processing-engine streaming streaming-analytics streaming-data streaming-data-pipelines streaming-data-processing webassembly
Language:Rust 5076
jitsu
jitsucom / jitsu
Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days
bigquery clickhouse data-collection data-connectors data-integration golang postgres redshift snowflake
Language:TypeScript 4574
rudder-server
rudderlabs / rudder-server
Privacy and Security focused Segment-alternative, in Golang and React
bigquery cdp customer-data customer-data-lake customer-data-pipeline customer-data-platform data-engineering data-integration data-pipeline data-synchronization data-warehouse elt etl event-streaming privacy redshift segment-alternative snowflake warehouse-management warehouse-native
Language:Go 4314
DTStack / chunjun
A data integration framework
flink bigdata data-integration framework java
Language:Java 4090
seandavi / awesome-single-cell
Community-curated list of software packages and data resources for single-cell, including RNA-seq, ATAC-seq, etc.
rna-seq-data gene-expression scrna-seq-data bioinformatics awesome-list dimensionality-reduction cell-cycle atac-seq cell-differentiation python clustering cell-populations gene-expression-profiles rna-seq-experiments cell-clusters analysis-pipeline data-integration data-visualization single-cell analysis
3552
bruin-data / ingestr
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
bigquery copy-database data-ingestion data-integration data-pipeline duckdb ingestion-pipeline mssql postgresql snowflake
Language:Python 3327
apache / incubator-devlake
Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
data data-analysis data-engineering data-integration data-transfers devops domain-layer etl golang integration jira open-source user-friendly dashboard-friendly dora hacktoberfest
Language:Go 2847
mara / mara-pipelines
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
data data-integration etl pipeline postgresql python
Language:Python 2085
bytedance / bitsail
BitSail is a distributed high-performance data integration engine which supports batch, streaming and incremental scenarios. BitSail is widely used to synchronize hundreds of trillions of data every day.
big-data data-integration data-lake data-pipeline data-synchronization flink high-performance real-time
Language:Java 1675
apache / hop
Hop Orchestration Platform
apache data-integration etl hop java orchestration pipeline streaming workflow
Language:Java 1235
heathersherry / Knowledge-Graph-Tutorials-and-Papers
Insightful Tutorials and Papers about Knowledge Graphs
entity-linking knowledge-base knowledge-graph mmkg kgqa awesome awesome-kg awesome-list information-extration kg knowledge-graph-embedding knowledge-graphs knowledge-graph-construction knowledge-graph-reasoning data-integration knowledge-graph-completion knowledge-graph-question-answering knowledge-graph-for-recommendation knowledge-graph-representation
962
kuwala
kuwala-io / kuwala
Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data science models and products with a focus on geospatial data. Currently, the following data connectors are available worldwide: a) High-resolution demographics data b) Point of Interests from Open Street Map c) Google Popular Times
data data-integration data-science open-data spatial-analysis elt kuwala open-source scraping dbt postgres pyspark python jupyter admin-boundaries google-trends population react no-code react-flow
Language:JavaScript 805
apache / seatunnel-web
SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).
apache data-integration data-pipeline etl-framework high-performance offline real-time seatunnel sql-engine
Language:Java 736
artie-labs / transfer
Database replication platform that leverages change data capture. Stream production data from databases to your data warehouse (Snowflake, BigQuery, Redshift, Databricks) in real-time.
snowflake cdc change-data-capture golang bigquery kafka apache-kafka data-integration data-pipelines database debezium elt redshift
Language:Go 680
immunogenomics / harmony
Fast, sensitive and accurate integration of single-cell data with Harmony
scrna-seq algorithm data-integration r
Language:R 600
saeyslab / nichenetr
NicheNet: predict active ligand-target links between interacting cells
single-cell-omics intercellular-communication ligand-receptor ligand-target data-integration single-cell-rna-seq rna-seq gene-expression network-inference cell-cell-communication
Language:R 586
conduit
ConduitIO / conduit
Conduit streams data between data stores. Kafka Connect replacement. No JVM required.
conduit data-engineering data-integration data-pipeline data-stream etl go kafka kafkaconnect
Language:Go 564
leesf / hudi-resources
汇总Apache Hudi相关资料
hudi apachehudi apache hudi-resources datalake bigdata stream-processing incremental-processing data-integration
557
pracdata / awesome-open-source-data-engineering
A curated list of open source tools used in analytics platforms and data engineering ecosystem
awesome awesome-list data-analytics data-engineering data-platform database self-hosted mlops data-storage data data-integration data-lakehouse datalake lakehouse workflow-engine analytics data-warehouse observability data-pipeline etl
389
theislab / scarches
Reference mapping for single-cell genomics
single-cell deep-learning rna-seq-analysis data-integration batch-correction multimodal-deep-learning multiomics single-cell-genomics scrna-seq human-cell-atlas
Language:Jupyter Notebook 387
graphform / swim-rust
Self-contained distributed software platform for building stateful, massively real-time streaming applications in Rust.
actor-model async decentralized-applications distributed-systems real-time rust stateful web data-integration framework kafka stream-processing streaming streaming-data-pipelines serverless
Language:Rust 351
recap
gabledata / recap
Work with your web service, database, and streaming schemas in a single format.
data-catalog metadata data-discovery recap data-engineering data-integration data-pipelines etl
Language:Python 343
CategoricalData / CQL
Categorical Query Language IDE
cql category-theory functional-programming data-migratation etl data-integration data-science data-exchange
Language:Java 319
elbwalker / walkerOS
Open-source data collection and tag management
data-collection privacy-by-design behavioral-data component-driven event-tracking web-analytics tagging tag-manager
Language:TypeScript 310
hetio / hetionet
Hetionet: an integrative network of disease
hetionet rephetio network neo4j hetnet drug-repurposing data-integration
Language:HTML 304
cuebook / cuelake
Use SQL to build ELT pipelines on a data lakehouse.
apache-iceberg delta lakehouse datalake data-lake elt etl data-engineering data-integration data-ingestion apache-spark spark-sql upsert incremental-updates data-transfer pipelines data-pipeline zeppelin-notebook sql
Language:JavaScript 288
CommonCoreOntology / CommonCoreOntologies
The Common Core Ontology Repository holds the current released version of the Common Core Ontology suite.
cco applied-ontology bfo semantics ontologies ontology-suite owl-ontology interoperability data-integration semantic-consistency
Language:Makefile 274