data-infrastructure

There are 6 repositories under data-infrastructure topic.

• zalando / postgres-operator

  Postgres operator creates and manages PostgreSQL clusters running in Kubernetes

  clusterdata-infrastructuredatabase-as-a-servicegolangkubernetesmanaged-servicesoperatorpostgrespostgres-operatorpostgresql
  Language:Go 4956

• CrunchyData / postgres-operator

  Production PostgreSQL for Kubernetes, from high availability Postgres clusters to full-scale database-as-a-service.

  data-infrastructuredatabasedatabase-as-a-servicedatabase-managementdisaster-recoveryhigh-availabilitykuberneteskubernetes-operatoroperatorpgopostgrespostgres-operatorpostgresqlpostgresql-clusterspostgresql-metricspostgresql-monitoring
  Language:Go 4304
• preswald

  StructuredLabs / preswald

  Preswald is a WASM packager for Python-based interactive data apps: bundle full complex data workflows, particularly visualizations, into single files, runnable completely in-browser, using Pyodide, DuckDB, Pandas, and Plotly, Matplotlib, etc. Build dashboards, reports, and notebooks that run offline, load fast, and share like a document.

  aianalyticsdataanalytics-engineeringcopilotdata-applicationsdata-infrastructuredata-pipelinesdata-sdkdata-visualizationgptllmopen-sourcepythonschema-managementvscode
  Language:Python 4301
• cocoindex

  cocoindex-io / cocoindex

  Data transformation framework for AI. Ultra performant, with incremental processing. 🌟 Star if you like it!

  aichange-data-capturecontext-engineeringdatadata-engineeringdata-indexingdata-infrastructuredata-processingetlhacktoberfesthelp-wantedindexingknowledge-graphllmpipelinepythonragreal-timerustsemantic-search
  Language:Rust 3254

• zalando / spilo

  Highly available elephant herd: HA PostgreSQL cluster using Docker

  data-infrastructuredockerdocker-imagehigh-availabilitypatronipostgresqlpython
  Language:Python 1742

• tensorbase / tensorbase

  TensorBase is a new big data warehousing with modern efforts.

  rustbigdatadatabaseanalyticsmoderninfrastructuredatadata-infrastructurehigh-performanceengineeringrust-langwarehousedata-warehouse
  Language:Rust 1458

• zalando / nakadi

  A distributed event bus that implements a RESTful API abstraction on top of Kafka-like queues

  apisdata-infrastructureevent-busjavajava-8kafkamicroservicespostgresqlrestful
  Language:Java 965

• zalando / PGObserver

  A battle-tested, flexible & comprehensive monitoring solution for your PostgreSQL databases

  data-infrastructuremonitoring
  Language:Python 315

• thedataengineeringbook / thedataengineeringbook

  The Data Engineering Book - หนังสือวิศวกรรมข้อมูล ของคนไทย เพื่อคนไทย

  data-engineeringdatahacktoberfestbookdata-engineerdata-pipelinedata-integrationdata-ingestiondata-infrastructure
  Language:JavaScript 114

• zalando-incubator / spark-json-schema

  JSON schema parser for Apache Spark

  data-infrastructure
  Language:Scala 82

• abhishek-ch / data-machinelearning-the-boring-way

  Build & Learn Data Engineering,Machine Learning over Kubernetes. No Shortcut approach.

  data-infrastructuredataengineeringdatasciencekubernetesmachine-learningmlops
  Language:Python 57

• zalando-nakadi / kanadi

  Kanadi is a Nakadi client for Scala

  nakadiscalaakka-streamsakka-streamkafkacircedata-infrastructurehacktoberfest
  Language:Scala 28
• opensnowcat-enrich

  opensnowcat / opensnowcat-enrich

  OpenSnowcat Enricher (Apache 2.0 License)

  analyticsevent-pipelinesnowplowdata-engineeringdata-pipelineai-data-collectionapache-2-0-licensebehavioral-datadata-infrastructuredata-sciencekafkakinesisopen-sourcewarpstreamdata-collectionmarketing-analyticsproduct-analyticssnowplow-eventssnowplow-pipeline
  Language:Scala 27

• aivanzhang / panda_patrol

  datadata-engineeringdata-infrastructuredata-monitoringdata-qualitypipelinespython
  Language:Python 23

• zalando-incubator / darty

  Data dependency manager

  data-infrastructure
  Language:Python 22

• bizzabo / elasticsearch_to_bigquery_data_pipeline

  A generic data pipeline which will map Elasticsearch documents to Bigquery table rows

  rnddata-infrastructure
  Language:Kotlin 14

• Jzbonner / dataengineering-db

  Information relating to topics on Data Engineering, Data Infrastructure, Data Storing, Data Warehouses and Business Analysis. For those interested in both conceptual theory and use case examples for database design and development.

  data-engineerdata-infrastructuredata-analysisdata-warehousesqlalchemy-databasesqlalchemysqlite3
  10

• alphagov / consent-api

  Service for sharing user consent to cookies across multiple domains

  cookie-consentdata-infrastructuredata-servicessdedata-infrastructure-team
  Language:TypeScript 8

• yennanliu / data_infra_repo

  Collections of POC/dev data infrastructure. | #SE

  data-engineeringdata-infrastructuredockerinfrastructurebackendjavadatabasebackend-servicesceleryredismysqlkafkapostgresqlelasticsearchsystem-designmongodbflowersupersetdeploymentdevops
  Language:Python 6

• anna-geller / kestra-terraform-examples

  Bring Infrastructure as Code best practices to your data workflows with Kestra and Terraform

  automationawsdata-engineeringdata-infrastructuredata-orchestrationinfrastructure-as-codekestraplatform-engineeringterraformworkflow-as-codeworkflow-orchestration
  Language:HCL 5

• alphagov / sde-prototype-govuk

  A fake GOV.UK homepage and start pages for SDE prototype services

  cptodata-infrastructuredata-infrastructure-teamdata-servicessde
  Language:HTML 4

• amkrajewski / mpdd-alignn

  MPDD Calculator for Atomistic Line Graph Neural Network Deployment

  computational-materials-sciencedata-infrastructuremachine-learningmodel-deploymentmpddphases-research-lab
  Language:Python 4

• Noobzik / ATL-Datamart

  TP d'architecture décisionnel à destination des étudiants de l'EPSI et DC Paris. Le but est de déployer une architecture data dès la récupération de la donnée vers la restitution sous la forme de dataviz en passant par un Datalake, Data Warehouse et d'un Data Mart

  data-infrastructuredata-warehousedatalakeetl-pipelineminio
  Language:Python 4

• ablange / nix-data-mesh

  A practical data mesh reference implementation, powered by open-source.

  bashdata-infrastructuredata-meshdata-platformkubernetesmakefilenixopen-source
  Language:Jinja 1

• alphagov / analytics-settings-database

  Export Google Analytics (GA4 and UA) settings

  cptodata-infrastructuregoogle-analytics
  Language:Python 1

• alphagov / sde-prototype-haas

  SDE prototype dummy service - Hexagrams as a Service

  cptodata-infrastructuredata-infrastructure-teamdata-servicessde
  Language:HTML 1

• ICPSR / mica-data-descriptor

  Processing code for Scientific Data Descriptor paper

  data-curationdata-infrastructuredata-reuse
  Language:Jupyter Notebook 1

• ilssaf / data-platform-deployer

  CLI tool for automatic data platform deployment

  cdcclickhousedata-engineeringdata-infrastructuredata-platformdevopsetlinfrastructure-as-codekafkakafka-connectpostgresqls3
  Language:Python 1

• k0rsakov / infrastructure_for_data_engineer_dbt

  Инфраструктура для Data-Engineer DBT

  data-infrastructuredbtdbt-coredbt-duckdbdbt-s3dbt-tutorialduckdbduckdb-exit-infrastructures3
  Language:Python 1

• LatiefDataVisionary / data-management-and-data-infrastructure-college-task

  data-infrastructuredata-managementdata-management-systeminfrastructure-datainfrastructure-data-warehouse
  Language:Jupyter Notebook 1

• smdoh / pipebird

  Open-source API to securely share data with customers.

  apidata-infrastructuredata-pipelines
  Language:TypeScript 1

• apelullo / yelp_health_data_curation_ops

  An AWS-based data pipeline to extract, process, store, and monitor Yelp "health-related" facility data in support of ongoing health system initiatives.

  academic-researchautomationawsdata-accessdata-curationdata-infrastructuredata-pipelineshealth-dataoperationsoperations-researchpythonyelp-dataset
  Language:Jupyter Notebook 0

• iTrauco / streaming-data-platform

  skeleton streaming data platform on gcp...

  big-datadatadata-engineeringdata-infrastructuredata-scienceengineeringgoogle-cloudplatform-engineeringpythonstreaming-data
  Language:Python 0

• Corey4005 / STEMNET-Daily-Files

  The purpose of this repository is to create a data infrastructure that will communicate with the STEMNET server at the University of Alabama Huntsville. In particular, the goal is to give anyone the capability to create clean daily files from all available stations on linux machines.

  data-infrastructure
  Language:Python

• maksimkurb / spilo

  Fork of Zalando Postgres Splio with pgvecto.rs and VectorChord extensions installed (Immich-compatible)

  data-infrastructuredockerdocker-imagehigh-availabilityimmichpatronipostgresq
  Language:Python

• seedcase-project / template-data-package

  An opinionated template for Data Packages built with Seedcase packages.

  copiercopier-templatedata-packagefrictionless-datatemplatedata-engineeringdata-infrastructurefair-datatemplate-projecttemplate-repositorytemplate-data-package
  Language:Jinja