apache-beam

There are 8 repositories under apache-beam topic.

• tensorflow / tfx

  TFX is an end-to-end platform for deploying production ML pipelines

  apache-beammachine-learningtensorflow
  Language:Python 2162

• GoogleCloudPlatform / DataflowTemplates

  Cloud Dataflow Google-provided templates for solving in-Cloud data tasks

  apache-beambigquerybigtabledataflow-templatesgoogle-cloud-dataflowgoogle-cloud-spannergoogle-cloud-storage
  Language:Java 1236

• nielsbasjes / yauaa

  Yet Another UserAgent Analyzer

  user-agentparseanalyzerjavahiveflinkapache-beamapache-flinkapache-hivenifi-processoruser-agent-parsernifi-processorssnowflakesnowplowsnowplowanalyticstrino-pluginuseragent-parseruseragentparserclient-hintsuser-agent-analysis
  Language:Java 854

• GoogleCloudPlatform / flink-on-k8s-operator

  [DEPRECATED] Kubernetes operator for managing the lifecycle of Apache Flink and Beam applications.

  google-cloud-dataprocflink-operatorkubernetes-operatorapache-beamapache-flinkkubernetesoperator
  Language:Go 658

• blockchain-etl / bitcoin-etl

  ETL scripts for Bitcoin, Litecoin, Dash, Zcash, Doge, Bitcoin Cash. Available in Google BigQuery https://goo.gl/oY5BCQ

  bitcoinetlgcpcryptocurrencylitecoindogecoindashbitcoincashblockchain-analyticscryptodata-analyticsdata-engineeringon-chain-analysisweb3zcashapache-beamgoogle-dataflowgoogle-pubsub
  Language:Python 421

• google / weather-tools

  Tools to make weather data accessible and useful.

  pythonapache-beamweather
  Language:Python 224

• spotify / flink-on-k8s-operator

  Kubernetes operator for managing the lifecycle of Apache Flink and Beam applications.

  apache-beamapache-flinkflinkflink-operatorkuberneteskubernetes-operator
  Language:Go 211

• google / fhir-data-pipes

  A collection of tools for extracting FHIR resources and analytics services on top of that data.

  analyticsetlfhir-storeapache-beamfhirparquet
  Language:Jupyter Notebook 190

• google / tensorflow-recorder

  TFRecorder makes it easy to create TensorFlow records (TFRecords) from Pandas DataFrames and CSVs files containing images or structured data.

  tensorflow-recordertfrecordstensorflowpreprocessingimage-processingapache-beam
  Language:Python 180

• ngrunwald / datasplash

  Clojure API for a more dynamic Google Dataflow

  apache-beamclojuredistributed-computinggoogle-cloudgoogle-dataflow
  Language:Clojure 131

• mohaseeb / beam-nuggets

  Collection of transforms for the Apache beam python SDK.

  apache-beamrelational-databasespython
  Language:Python 90

• blockchain-etl / blockchain-etl-streaming

  Streaming Ethereum and Bitcoin blockchain data to Google Pub/Sub or Postgres in Kubernetes

  real-timebitcoinethereumcryptocurrencyapache-beamblockchainblockchain-analyticscryptodata-analyticsdata-engineeringetlgcpgoogle-bigquerygoogle-dataflowgoogle-pubsubon-chain-analysisreal-time-analyticsstream-processingweb3google-cloud-platform
  Language:Python 80
• asgarde

  tosun-si / asgarde

  Asgarde allows simplifying error handling with Apache Beam Java, with less code, more concise and expressive code.

  apache-beamcloud-dataflowerror-handlinggoogle-cloud-platformjavakotlin
  Language:Java 74

• mercari / DataflowTemplate

  Mercari Dataflow Template

  apache-beamgoogle-cloudcloud-dataflow
  Language:Java 72

• Fematich / mlengine-boilerplate

  Repository to quickly get you started with new Machine Learning projects on Google Cloud Platform. More info(slides):

  apache-beamtensorflowml-enginegoogle-cloud-platformgoogle-cloud-dataflowboilerplate
  Language:Python 64

• yu-iskw / bigquery-to-datastore

  Export a whole BigQuery table to Google Datastore with Apache Beam/Google Dataflow

  google-datastorebigquerygoogle-cloudgoogle-dataflowapache-beambeam
  Language:Java 58

• xmlking / micro-apps

  Microservices in Post-Kubernetes Era. A polyglot monorepo

  kotlingradle-kotlin-dslgitopsgitflowjenkinsgitflow-workflowsemantic-releasesemantic-versioningconventional-commitsconventional-changelogmicroservicequarkusiomicronautsprintbootmonorepoapache-beam
  Language:Kotlin 53

• luisbelloch / data_processing_course

  Some class materials for a data processing course using PySpark

  apache-beambigdatacoursedata-processingpysparksparkstream-processing
  Language:Python 52

• blockchain-etl / blockchain-etl-architecture

  Blockchain ETL Architecture

  apache-beamdata-analyticsdata-engineeringgcpgkegoogle-bigquerygoogle-cloudgoogle-cloud-platformgoogle-container-enginegoogle-dataflowgoogle-pubsubkubernetesblockchain-analyticsethereumon-chain-analysisblockchaincryptocryptocurrencyreal-time-analyticsetl
  47

• doitintl / banias

  Opinionated serverless event analytics pipeline

  analyticsapache-beambigdatadataflowgolang
  Language:Go 43

• tosun-si / pasgarde

  Asgarde allows simplifying error handling with Apache Beam Python, with less code, more concise and expressive code.

  apache-beamcloud-dataflowerror-handlinggoogle-cloud-platformpython
  Language:Python 31

• asaharland / beam-pipeline-examples

  Apache Beam examples for running on Google Cloud Dataflow.

  apache-beamaws-s3google-cloudgoogle-cloud-dataflowgoogle-cloud-platformgoogle-cloud-pubsubgoogle-cloud-storage
  Language:Java 30

• NucleusEngineering / hack-your-pipe

  Efficient streaming data ingestion, transformation & activation

  cloud-rundataflowpubsubvertex-aidata-engineeringstreaming-datadata-opsml-opsapache-beamgoogle-cloud-platform
  Language:Python 28

• ingestionflow / pysql-beam

  pythonapache-beamgcpdataflowmysqlpostgresqlmssql
  Language:Python 25

• mozilla-services / foxsec-pipeline

  Log analysis pipeline utilizing Apache Beam

  apache-beamlog-analysissecuritydataflow
  Language:Java 25

• sayakpaul / count-tokens-hf-datasets

  This project shows how to derive the total number of training tokens from a large text dataset from 🤗 datasets with Apache Beam and Dataflow.

  apache-beamdataflowhf-datasetstokenizerstransformersunigram-tokenization
  Language:Python 25

• mercari / DataflowTemplates

  Convenient Dataflow pipelines for transforming data between cloud data sources

  apache-beamdataflowdataflow-templatesspannerbigquery
  Language:Java 24

• google-parfait / dataset_grouper

  Libraries for efficient and scalable group-structured dataset pipelines.

  apache-beamdatasetsfederated-learningjaxpytorchtensorflowtensorflow-datasets
  Language:Python 23

• google / consent-based-conversion-adjustments

  Code to statistically up-weight conversion values of consenting customers to feed up to 100% of the factual conversion values back into Google Ads.

  apache-beamgoogle-adsgoogle-cloud-platform
  Language:Python 22

• janaom / gcp-data-engineering-etl-with-composer-dataflow

  This project leverages GCS, Composer, Dataflow, BigQuery, and Looker on Google Cloud Platform (GCP) to build a robust data engineering solution for processing, storing, and reporting daily transaction data in the online food delivery industry.

  airflowapache-beamcloud-composercloud-storagedata-engineeringdataflowde-projectgcpgcslooker
  Language:Python 22

• datastacktv / apache-beam-explained

  Source code for the YouTube video, Apache Beam Explained in 12 Minutes

  apache-beam
  Language:Python 21

• delftdata / stateflow

  Prototype which extracts stateful dataflows by analysing Python code.

  apache-beamapache-flinkawsaws-lambdafastapikafkakinesispythonstateflow
  Language:Python 21

• esakik / beam-mysql-connector

  An Apache Beam I/O connector for seamless integration with MySQL database 🔗 https://beam.apache.org/documentation/io/connectors/#other-io-connectors-for-apache-beam

  apache-beammysqlpypipython
  Language:Python 21

• O2-Czech-Republic / proxima-platform

  The Proxima platform.

  batch-processinganalytical-platformunified-data-processingapache-sparkapache-flinkapache-beamiot-platformdata-meshstream-processing
  Language:Java 21

• carted / processing-text-data

  Presents an optimized Apache Beam pipeline for generating sentence embeddings (runnable on Cloud Dataflow).

  tensorflowapache-beamdataflowtext-databertuse-berttfhub
  Language:Python 20

• cassiobolba / Data-Engineering

  Projects and studies regarding Data Engineering Area

  pythonpysparksqlgitgitcilambda-functionsapache-beam
  Language:HTML 20