data-infrastructure

There are 5 repositories under data-infrastructure topic.

• zalando / postgres-operator

  Postgres operator creates and manages PostgreSQL clusters running in Kubernetes

  clusterdata-infrastructuredatabase-as-a-servicegolangkubernetesmanaged-servicesoperatorpostgrespostgres-operatorpostgresql
  Language:Go 4001

• CrunchyData / postgres-operator

  Production PostgreSQL for Kubernetes, from high availability Postgres clusters to full-scale database-as-a-service.

  postgresqlkubernetesoperatorpostgrespostgres-operatorhigh-availabilitydatabasedatabase-managementpostgresql-clustersdisaster-recoverypostgresql-monitoringpostgresql-metricsdatabase-as-a-servicedata-infrastructurekubernetes-operatorpgo
  Language:Go 3744

• tensorbase / tensorbase

  TensorBase is a new big data warehousing with modern efforts.

  rustbigdatadatabaseanalyticsmoderninfrastructuredatadata-infrastructurehigh-performanceengineeringrust-langwarehousedata-warehouse
  Language:Rust 1429

• zalando / spilo

  Highly available elephant herd: HA PostgreSQL cluster using Docker

  data-infrastructuredockerdocker-imagehigh-availabilitypatronipostgresqlpython
  Language:Python 1326

• zalando / nakadi

  A distributed event bus that implements a RESTful API abstraction on top of Kafka-like queues

  javaapisevent-busrestfulmicroservicespostgresqlkafkajava-8data-infrastructure
  Language:Java 949

• zalando / PGObserver

  A battle-tested, flexible & comprehensive monitoring solution for your PostgreSQL databases

  monitoringdata-infrastructure
  Language:Python 315

• uktrade / stream-unzip

  Python function to stream unzip all the files in a ZIP archive on the fly

  data-infrastructurezipdiapp
  Language:Python 252

• uktrade / mbtiles-s3-server

  Python server to on-the-fly extract and serve vector tiles from an mbtiles file on S3

  data-infrastructure
  Language:Python 136

• uktrade / sqlite-s3vfs

  Python writable virtual filesystem for SQLite on S3

  data-infrastructurediapp
  Language:Python 112

• thedataengineeringbook / thedataengineeringbook

  The Data Engineering Book - หนังสือวิศวกรรมข้อมูล ของคนไทย เพื่อคนไทย

  data-engineeringdatahacktoberfestbookdata-engineerdata-pipelinedata-integrationdata-ingestiondata-infrastructure
  Language:JavaScript 103

• uktrade / stream-zip

  Python function to construct a ZIP archive on the fly

  data-infrastructurezipdiapp
  Language:Python 88

• zalando-incubator / spark-json-schema

  JSON schema parser for Apache Spark

  data-infrastructure
  Language:Scala 81

• abhishek-ch / data-machinelearning-the-boring-way

  Build & Learn Data Engineering,Machine Learning over Kubernetes. No Shortcut approach.

  data-infrastructuredataengineeringdatasciencekubernetesmachine-learningmlops
  Language:Python 52

• uktrade / mobius3

  Continuously sync folder to S3, using inotify under the hood

  data-infrastructuredata-workspace
  Language:Python 47

• uktrade / fargatespawner

  Spawns JupyterHub single user servers in Docker containers running in AWS Fargate

  data-infrastructure
  Language:Python 44

• uktrade / data-workspace-frontend

  An open source data analysis platform with features for users with a range of technical skills

  data-infrastructuredata-workspace
  Language:Python 43

• uktrade / pg-bulk-ingest

  Python utility function to ingest data into a SQLAlchemy-defined PostgreSQL table

  postgresqlpythondata-infrastructurediapp
  Language:Python 34

• uktrade / dns-rewrite-proxy

  A DNS proxy server that conditionally rewrites and filters A record requests

  data-infrastructuredata-workspace
  Language:Python 30

• zalando-nakadi / kanadi

  Kanadi is a Nakadi client for Scala

  nakadiscalaakka-streamsakka-streamkafkacircedata-infrastructurehacktoberfest
  Language:Scala 30

• uktrade / stream-sqlite

  Python function to extract rows from a SQLite file while iterating over its bytes

  sqlitedata-infrastructure
  Language:Python 23

• uktrade / tidy-json-to-csv

  Convert JSON to a set of tidy CSV files

  data-infrastructurediapp
  Language:Python 22

• zalando-incubator / darty

  Data dependency manager

  data-infrastructure
  Language:Python 22

• aivanzhang / panda_patrol

  datadata-engineeringdata-infrastructuredata-monitoringdata-qualitypipelinespython
  Language:Python 21

• uktrade / jupyters3

  Jupyter Notebook Contents Manager for AWS S3

  data-infrastructure
  Language:Python 17

• bizzabo / elasticsearch_to_bigquery_data_pipeline

  A generic data pipeline which will map Elasticsearch documents to Bigquery table rows

  rnddata-infrastructure
  Language:Kotlin 15

• uktrade / stream-read-xbrl

  Python package to parse Companies House accounts data in a streaming way

  data-infrastructurediapp
  Language:Python 13

• Jzbonner / dataengineering-db

  Information relating to topics on Data Engineering, Data Infrastructure, Data Storing, Data Warehouses and Business Analysis. For those interested in both conceptual theory and use case examples for database design and development.

  data-engineerdata-infrastructuredata-analysisdata-warehousesqlalchemy-databasesqlalchemysqlite3
  9

• uktrade / streampq

  Python PostgreSQL adapter to stream results of multi-statement queries without a server-side cursor

  postgresqlpythondata-infrastructure
  Language:Python 8

• alphagov / consent-api

  Service for sharing user consent to cookies across multiple domains

  cookie-consentdata-infrastructuredata-servicessdedata-infrastructure-team
  Language:Python 7

• uktrade / iterable-subprocess

  Python context manager to communicate with a subprocess using iterables: for when data is too big to fit in memory and has to be streamed

  data-infrastructurediapp
  Language:Python 7

• uktrade / to-file-like-obj

  Python utility function to convert an iterable of bytes or str to a readable file-like object

  data-infrastructurediapp
  Language:Python 7

• uktrade / jwt-postgresql-proxy

  Stateless JWT authentication in front of PostgreSQL

  data-infrastructure
  Language:Python 6

• yennanliu / data_infra_repo

  Collections of POC/dev data infrastructure. | #SE

  data-engineeringdata-infrastructuredockerinfrastructurebackendjavadatabasebackend-servicesceleryredismysqlkafkapostgresqlelasticsearchsystem-designmongodbflowersupersetdeploymentdevops
  Language:Python 6

• anna-geller / kestra-terraform-examples

  Bring Infrastructure as Code best practices to your data workflows with Kestra and Terraform

  automationawsdata-engineeringdata-infrastructuredata-orchestrationinfrastructure-as-codekestraplatform-engineeringterraformworkflow-as-codeworkflow-orchestration
  Language:HCL 5

• uktrade / public-data-api

  The source for the Department for International Trade's Public Data API

  data-infrastructurediapp
  Language:HTML 5

• uktrade / streamlit-gov-uk-components

  A collection of Streamlit components that use or are inspired by the GOV.UK Design System

  data-infrastructure
  Language:Shell 5