apache-beam

There are 8 repositories under apache-beam topic.

• tensorflow / tfx

  TFX is an end-to-end platform for deploying production ML pipelines

  apache-beammachine-learningtensorflow
  Language:Python 2082

• GoogleCloudPlatform / DataflowTemplates

  Cloud Dataflow Google-provided templates for solving in-Cloud data tasks

  apache-beamdataflow-templatesgoogle-cloud-dataflowgoogle-cloud-storagegoogle-cloud-spannerbigquerybigtable
  Language:Java 1099

• nielsbasjes / yauaa

  Yet Another UserAgent Analyzer

  user-agentparseanalyzerjavahiveflinkapache-beamapache-flinkapache-hivenifi-processoruser-agent-parsernifi-processorssnowflakesnowplowsnowplowanalyticstrino-pluginuseragent-parseruseragentparserclient-hintsuser-agent-analysis
  Language:Java 735

• GoogleCloudPlatform / flink-on-k8s-operator

  [DEPRECATED] Kubernetes operator for managing the lifecycle of Apache Flink and Beam applications.

  google-cloud-dataprocflink-operatorkubernetes-operatorapache-beamapache-flinkkubernetesoperator
  Language:Go 658

• blockchain-etl / bitcoin-etl

  ETL scripts for Bitcoin, Litecoin, Dash, Zcash, Doge, Bitcoin Cash. Available in Google BigQuery https://goo.gl/oY5BCQ

  bitcoinetlgcpcryptocurrencylitecoindogecoindashbitcoincashblockchain-analyticscryptodata-analyticsdata-engineeringon-chain-analysisweb3zcashapache-beamgoogle-dataflowgoogle-pubsub
  Language:Python 388

• google / weather-tools

  Tools to make weather data accessible and useful.

  pythonapache-beamweather
  Language:Python 199

• google / tensorflow-recorder

  TFRecorder makes it easy to create TensorFlow records (TFRecords) from Pandas DataFrames and CSVs files containing images or structured data.

  tensorflow-recordertfrecordstensorflowpreprocessingimage-processingapache-beam
  Language:Python 180

• spotify / flink-on-k8s-operator

  Kubernetes operator for managing the lifecycle of Apache Flink and Beam applications.

  kubernetesapache-beamapache-flinkkubernetes-operatorflink-operatorflink
  Language:Go 180

• google / fhir-data-pipes

  A collection of tools for extracting FHIR resources and analytics services on top of that data.

  analyticsetlfhir-storeapache-beamfhirparquet
  Language:Java 130

• ngrunwald / datasplash

  Clojure API for a more dynamic Google Dataflow

  apache-beamclojuredistributed-computinggoogle-cloudgoogle-dataflow
  Language:Clojure 130

• mohaseeb / beam-nuggets

  Collection of transforms for the Apache beam python SDK.

  apache-beamrelational-databasespython
  Language:Python 86

• blockchain-etl / blockchain-etl-streaming

  Streaming Ethereum and Bitcoin blockchain data to Google Pub/Sub or Postgres in Kubernetes

  real-timebitcoinethereumcryptocurrencyapache-beamblockchainblockchain-analyticscryptodata-analyticsdata-engineeringetlgcpgoogle-bigquerygoogle-dataflowgoogle-pubsubon-chain-analysisreal-time-analyticsstream-processingweb3google-cloud-platform
  Language:Python 74
• asgarde

  tosun-si / asgarde

  Asgarde allows simplifying error handling with Apache Beam Java, with less code, more concise and expressive code.

  apache-beamcloud-dataflowerror-handlinggoogle-cloud-platformjavakotlin
  Language:Java 71

• Fematich / mlengine-boilerplate

  Repository to quickly get you started with new Machine Learning projects on Google Cloud Platform. More info(slides):

  apache-beamtensorflowml-enginegoogle-cloud-platformgoogle-cloud-dataflowboilerplate
  Language:Python 62

• mercari / DataflowTemplate

  Mercari Dataflow Template

  apache-beamgoogle-cloudcloud-dataflow
  Language:Java 62

• yu-iskw / bigquery-to-datastore

  Export a whole BigQuery table to Google Datastore with Apache Beam/Google Dataflow

  google-datastorebigquerygoogle-cloudgoogle-dataflowapache-beambeam
  Language:Java 58

• luisbelloch / data_processing_course

  Some class materials for a data processing course using PySpark

  apache-beambigdatacoursedata-processingpysparksparkstream-processing
  Language:Python 52

• xmlking / micro-apps

  Microservices in Post-Kubernetes Era. A polyglot monorepo

  kotlingradle-kotlin-dslgitopsgitflowjenkinsgitflow-workflowsemantic-releasesemantic-versioningconventional-commitsconventional-changelogmicroservicequarkusiomicronautsprintbootmonorepoapache-beam
  Language:Kotlin 51

• blockchain-etl / blockchain-etl-architecture

  Blockchain ETL Architecture

  apache-beamdata-analyticsdata-engineeringgcpgkegoogle-bigquerygoogle-cloudgoogle-cloud-platformgoogle-container-enginegoogle-dataflowgoogle-pubsubkubernetesblockchain-analyticsethereumon-chain-analysisblockchaincryptocryptocurrencyreal-time-analyticsetl
  43

• doitintl / banias

  Opinionated serverless event analytics pipeline

  analyticsbigdatagolangdataflowapache-beam
  Language:Go 43

• asaharland / beam-pipeline-examples

  Apache Beam examples for running on Google Cloud Dataflow.

  google-cloudgoogle-cloud-platformgoogle-cloud-storagegoogle-cloud-dataflowaws-s3apache-beamgoogle-cloud-pubsub
  Language:Java 29

• tosun-si / pasgarde

  Asgarde allows simplifying error handling with Apache Beam Python, with less code, more concise and expressive code.

  apache-beamcloud-dataflowerror-handlinggoogle-cloud-platformpython
  Language:Python 29

• NucleusEngineering / hack-your-pipe

  Efficient streaming data ingestion, transformation & activation

  cloud-rundataflowpubsubvertex-aidata-engineeringstreaming-datadata-opsml-opsapache-beamgoogle-cloud-platform
  Language:Python 26

• mercari / DataflowTemplates

  Convenient Dataflow pipelines for transforming data between cloud data sources

  apache-beamdataflowdataflow-templatesspannerbigquery
  Language:Java 25

• mozilla-services / foxsec-pipeline

  Log analysis pipeline utilizing Apache Beam

  apache-beamlog-analysissecuritydataflow
  Language:Java 25

• pysql-beam-org / pysql-beam

  pythonapache-beamgcpdataflowmysqlpostgresqlmssql
  Language:Python 24

• sayakpaul / count-tokens-hf-datasets

  This project shows how to derive the total number of training tokens from a large text dataset from 🤗 datasets with Apache Beam and Dataflow.

  apache-beamdataflowtokenizerstransformershf-datasetsunigram-tokenization
  Language:Python 22

• esakik / beam-mysql-connector

  Apache Beam I/O connector designed for accessing MySQL databases.

  pythonapache-beammysqlpypi
  Language:Python 21

• google-parfait / dataset_grouper

  Libraries for efficient and scalable group-structured dataset pipelines.

  apache-beamdatasetsfederated-learningjaxpytorchtensorflowtensorflow-datasets
  Language:Python 21

• carted / processing-text-data

  Presents an optimized Apache Beam pipeline for generating sentence embeddings (runnable on Cloud Dataflow).

  tensorflowapache-beamdataflowtext-databertuse-berttfhub
  Language:Python 20

• datastacktv / apache-beam-explained

  Source code for the YouTube video, Apache Beam Explained in 12 Minutes

  apache-beam
  Language:Python 20

• cassiobolba / Data-Engineering

  Projects and studies regarding Data Engineering Area

  pythonpysparksqlgitgitcilambda-functionsapache-beam
  Language:HTML 19

• google / consent-based-conversion-adjustments

  Code to statistically up-weight conversion values of consenting customers to feed up to 100% of the factual conversion values back into Google Ads.

  apache-beamgoogle-adsgoogle-cloud-platform
  Language:Python 19

• O2-Czech-Republic / proxima-platform

  The Proxima platform.

  batch-processinganalytical-platformunified-data-processingapache-sparkapache-flinkapache-beamiot-platformdata-meshstream-processing
  Language:Java 18

• datastacktv / apache-beam-batch-processing

  Public source code for the Batch Processing with Apache Beam (Python) online course

  apache-beamcloud-dataflow
  Language:Python 17

• delftdata / stateflow

  Prototype which extracts stateful dataflows by analysing Python code.

  pythonkafkafastapistateflowawsaws-lambdaapache-flinkapache-beamkinesis
  Language:Python 17