data-infrastructure

There are 6 repositories under data-infrastructure topic.

• zalando / postgres-operator

  Postgres operator creates and manages PostgreSQL clusters running in Kubernetes

  kubernetespostgresqloperatorgolangpostgresclustermanaged-servicesdata-infrastructurepostgres-operatordatabase-as-a-service
  Language:Go 4862
• preswald

  StructuredLabs / preswald

  Preswald is a WASM packager for Python-based interactive data apps: bundle full complex data workflows, particularly visualizations, into single files, runnable completely in-browser, using Pyodide, DuckDB, Pandas, and Plotly, Matplotlib, etc. Build dashboards, reports, and notebooks that run offline, load fast, and share like a document.

  aianalyticsanalytics-engineeringcopilotdatadata-applicationsdata-infrastructuredata-pipelinesdata-sdkdata-visualizationgptllmopen-sourcepythonschema-managementvscode
  Language:Python 4312

• CrunchyData / postgres-operator

  Production PostgreSQL for Kubernetes, from high availability Postgres clusters to full-scale database-as-a-service.

  data-infrastructuredatabasedatabase-as-a-servicedatabase-managementdisaster-recoveryhigh-availabilitykuberneteskubernetes-operatoroperatorpgopostgrespostgres-operatorpostgresqlpostgresql-clusterspostgresql-metricspostgresql-monitoring
  Language:Go 4257
• cocoindex

  cocoindex-io / cocoindex

  Data transformation framework for AI. Ultra performant, with incremental processing.

  aichange-data-capturedatadata-indexingetlindexingpipelinepythonragreal-timerustsemantic-searchdata-engineeringdata-infrastructuredata-processingdataflowhelp-wantedknowledge-graphllmcontext-engineering
  Language:Rust 2806

• zalando / spilo

  Highly available elephant herd: HA PostgreSQL cluster using Docker

  data-infrastructuredockerdocker-imagehigh-availabilitypatronipostgresqlpython
  Language:Python 1722

• tensorbase / tensorbase

  TensorBase is a new big data warehousing with modern efforts.

  rustbigdatadatabaseanalyticsmoderninfrastructuredatadata-infrastructurehigh-performanceengineeringrust-langwarehousedata-warehouse
  Language:Rust 1453

• zalando / nakadi

  A distributed event bus that implements a RESTful API abstraction on top of Kafka-like queues

  javaapisevent-busrestfulmicroservicespostgresqlkafkajava-8data-infrastructure
  Language:Java 965

• zalando / PGObserver

  A battle-tested, flexible & comprehensive monitoring solution for your PostgreSQL databases

  data-infrastructuremonitoring
  Language:Python 315

• uktrade / stream-unzip

  Python function to stream unzip all the files in a ZIP archive on the fly

  data-infrastructurediappzip
  Language:Python 291

• uktrade / sqlite-s3vfs

  Python writable virtual filesystem for SQLite on S3

  data-infrastructurediapp
  Language:Python 175

• uktrade / mbtiles-s3-server

  Python server to on-the-fly extract and serve vector tiles from an mbtiles file on S3

  data-infrastructurediapp
  Language:Python 154

• uktrade / stream-zip

  Python function to construct a ZIP archive on the fly

  data-infrastructurediappzip
  Language:Python 133

• thedataengineeringbook / thedataengineeringbook

  The Data Engineering Book - หนังสือวิศวกรรมข้อมูล ของคนไทย เพื่อคนไทย

  data-engineeringdatahacktoberfestbookdata-engineerdata-pipelinedata-integrationdata-ingestiondata-infrastructure
  Language:JavaScript 112

• zalando-incubator / spark-json-schema

  JSON schema parser for Apache Spark

  data-infrastructure
  Language:Scala 81

• abhishek-ch / data-machinelearning-the-boring-way

  Build & Learn Data Engineering,Machine Learning over Kubernetes. No Shortcut approach.

  data-infrastructuredataengineeringdatasciencekubernetesmachine-learningmlops
  Language:Python 57

• uktrade / mobius3

  Continuously sync folder to S3, using inotify under the hood

  data-infrastructuredata-workspace
  Language:Python 55

• uktrade / fargatespawner

  Spawns JupyterHub single user servers in Docker containers running in AWS Fargate

  data-infrastructure
  Language:Python 48

• uktrade / data-workspace-frontend

  An open source data analysis platform with features for users with a range of technical skills

  data-infrastructuredata-workspacedata-workspace-frontend
  Language:Python 46

• uktrade / pg-bulk-ingest

  Python utility function to ingest data into a SQLAlchemy-defined PostgreSQL table

  postgresqlpythondata-infrastructurediapp
  Language:Python 37

• uktrade / dns-rewrite-proxy

  A DNS proxy server that conditionally rewrites and filters A record requests

  data-infrastructuredata-workspacediapp
  Language:Python 30

• zalando-nakadi / kanadi

  Kanadi is a Nakadi client for Scala

  nakadiscalaakka-streamsakka-streamkafkacircedata-infrastructurehacktoberfest
  Language:Scala 28

• uktrade / tidy-json-to-csv

  Convert JSON to a set of tidy CSV files

  data-infrastructurediapp
  Language:Python 23

• uktrade / stream-read-xbrl

  Python package to parse Companies House accounts data in a streaming way

  data-infrastructurediapp
  Language:Python 22

• uktrade / stream-sqlite

  Python function to extract rows from a SQLite file while iterating over its bytes

  sqlitedata-infrastructurediapp
  Language:Python 22

• zalando-incubator / darty

  Data dependency manager

  data-infrastructure
  Language:Python 22

• aivanzhang / panda_patrol

  datadata-engineeringdata-infrastructuredata-monitoringdata-qualitypipelinespython
  Language:Python 21

• uktrade / jupyters3

  Jupyter Notebook Contents Manager for AWS S3

  data-infrastructure
  Language:Python 18

• bizzabo / elasticsearch_to_bigquery_data_pipeline

  A generic data pipeline which will map Elasticsearch documents to Bigquery table rows

  rnddata-infrastructure
  Language:Kotlin 14

• uktrade / to-file-like-obj

  Python utility function to convert an iterable of bytes or str to a readable file-like object

  data-infrastructurediapp
  Language:Python 13

• Jzbonner / dataengineering-db

  Information relating to topics on Data Engineering, Data Infrastructure, Data Storing, Data Warehouses and Business Analysis. For those interested in both conceptual theory and use case examples for database design and development.

  data-engineerdata-infrastructuredata-analysisdata-warehousesqlalchemy-databasesqlalchemysqlite3
  10

• alphagov / consent-api

  Service for sharing user consent to cookies across multiple domains

  cookie-consentdata-infrastructuredata-servicessdedata-infrastructure-team
  Language:TypeScript 8

• uktrade / jwt-postgresql-proxy

  Stateless JWT authentication in front of PostgreSQL

  data-infrastructurediapp
  Language:Python 8

• uktrade / streampq

  Python PostgreSQL adapter to stream results of multi-statement queries without a server-side cursor

  postgresqlpythondata-infrastructurediapp
  Language:Python 8

• uktrade / iterable-subprocess

  Python context manager to communicate with a subprocess using iterables: for when data is too big to fit in memory and has to be streamed

  data-infrastructurediapp
  Language:Python 7

• uktrade / privacy-crawler

  An extension for Google Chrome that crawls a website for cookies and fingerprinting behaviour

  data-infrastructurediapp
  Language:JavaScript 7

• yennanliu / data_infra_repo

  Collections of POC/dev data infrastructure. | #SE

  data-engineeringdata-infrastructuredockerinfrastructurebackendjavadatabasebackend-servicesceleryredismysqlkafkapostgresqlelasticsearchsystem-designmongodbflowersupersetdeploymentdevops
  Language:Python 6