big-data-processing

There are 10 repositories under big-data-processing topic.

• drshahizan / BDM

  Course covers big data fundamentals, processes, technologies, platform ecosystem, and management for practical application development.

  big-databig-data-analyticsbig-data-architecturebig-data-processing
  Language:Jupyter Notebook 58
• dlt-with-debug

  souvik-databricks / dlt-with-debug

  A lightweight helper utility which allows developers to do interactive pipeline development by having a unified source code for both DLT run and Non-DLT interactive notebook run.

  big-databig-data-processingdatabricksdelta-live-tablesdltetletl-pipelinepython3spark
  Language:Python 49

• airscholar / FlinkCommerce

  This repository contains an Apache Flink application for real-time sales analytics built using Docker Compose to orchestrate the necessary infrastructure components, including Apache Flink, Elasticsearch, and Postgres

  apache-flinkbig-databig-data-processingpythonrealtime-streaming
  Language:Java 44
• eskimo

  eskimo-sh / eskimo

  Eskimo is a state of the art Big Data Infrastructure and Management Web Console to build, manage and operate Big Data 2.0 Analytics clusters on Kubernetes. This is the git repository of Eskimo Community Edition.

  bigdatabig-data-platformbig-databig-data-projectsbig-data-analyticsbig-data-processingwebconsolecluster-managementsparkelasticsearchkafkaglusterglusterfszeppelincerebrokibanaflinkkuberneteskubernetes-clusterkubernetes-setup
  Language:Java 26

• felipefrizzo / terraform-aws-kinesis-firehose

  This code creates a Kinesis Firehose in AWS to send CloudWatch log data to S3.

  terraformterraform-awsterraform-providerkinesis-firehosecloudwatch-logsparquetbig-databig-data-processingetl-jobanalytics
  Language:HCL 26

• StarPlatinumStudio / Flink-SQL-Practice

  Flink SQL 实战 -中文博客专栏

  apache-flinkstream-processingbig-data-processingsql
  Language:Java 16

• giucris / yasp

  Yet Another SPark Framework

  big-databig-data-processingeltetletl-frameworketl-pipelineframeworkscalasparksparksql
  Language:Scala 10

• pyajs / veronica

  big data processing and machine learning platform，just like useing sql

  big-data-processingmachine-learning-platformpysparkpython3sqlxql
  Language:Python 10

• impresso / impresso-text-acquisition

  🛠️ Python library to import OCR data in various formats into the canonical JSON format defined by the Impresso project.

  historical-newspapersbig-data-processingimpresso-project
  Language:Jupyter Notebook 9

• anjijava16 / GCP_Data_Enginner_Utils

  GCP_Data_Enginner

  bigquerybig-data-processinggcpgcp-storagepubsubdataflowpythonscaladataprocdataproc-clustershell-scriptpysparknotebook
  Language:Shell 8

• hope-data-science / R4BD

  R for Big Data (Chinese Version)

  big-datarbig-data-analytics-techniquesbig-data-processing
  Language:R 8

• bdnf / BigData-Engineering-Projects

  Data modeling with Cassandra, building Data Warehouse using Redshift and creation of Data Lake using Spark and Airflow

  airflowbig-data-analyticsbig-data-processingcassandradata-lakedata-warehouseredshiftspark
  Language:Jupyter Notebook 6

• tabletop-labs / tabletop

  A curated selection of tools, libraries and services that help tame your dataflow to productively build ambitious, data driven & reactive applications on a streaming lakehouse

  elasticscalingkafkastream-processingbig-databig-data-analyticsbig-data-processingmodern-data-stackreal-timemicroservicestimetravelsemi-structured-cloud-warehouse
  Language:Go 5

• theGuyWithBlackTie / electricChargingStations

  big-dataelectric-vehiclescharging-stationsspark-mlbig-data-processing
  Language:Jupyter Notebook 4

• vvittis / FlinkSampling

  Reservoir Sampling for Group-By Queries in Flink Platform. Answering effectively Single Aggregate.

  stratumstreaming-tuplesreservoir-samplingtopicjavastreaming-databig-data-analyticsbig-data-processingapache-flinkgroup-bysampling
  Language:Java 4

• akardapolov / dimension-db

  Hybrid time-series and block-column storage database engine written in Java

  big-data-processingcolumn-storedbmsjavasqltime-series
  Language:Java 3

• chandnii7 / Big-Data-Processing-Pipeline

  A pipeline that consumes twitter data to extract meaningful insights about a variety of topics using the following technologies: twitter API, Kafka, MongoDB, and Tableau.

  twitter-apikafkakafka-producerkafka-consumerkafka-streamingzookeepermongodbtableaubig-databig-data-processingdata-processing-pipelinesdata-analyticsnosql-databasedata-visualization
  Language:Python 3
• Big-Data-Algorithms

  kochlisGit / Big-Data-Algorithms

  Implementation of algorithms for big data using python, numpy, pandas.

  lshlsh-algorithmmin-hasingshinglingpythonsimilar-itemspcya-priorimultistage-pcymultihash-pcyfrequent-itemsetsfrequent-itemset-miningbloom-filterstream-miningstreamsbig-data-processing
  Language:Python 3

• mtumilowicz / big-data-scala-spark-batch-workshop

  Introduction to Spark Batch processing.

  sparkspark-sqlbatch-processingbig-databig-data-processingworkshopworkshop-materials
  Language:Scala 3

• software-competence-center-hagenberg / AVUBDI

  Github Repository for a versatile usable Big Data infrastructure (AVUBDI)

  kafkasparkdocker-composedockerbig-data-processingbig-data-platformdocker-swarmtemplate-project
  Language:Shell 3

• VladOnMyOwn / ctr-poisson-bootstrap

  Here I demonstrate the performance difference between the Poisson and the classic bootstrap by estimating the confidence interval for the difference of CTRs of the two user groups

  ab-testab-testingab-testsbootstrapstatistical-testsstatisticspoisson-bootstrapbig-databig-data-processingpythonclick-through-rate
  Language:Jupyter Notebook 3

• Anirban166 / Big-Data-ft.-Genomics

  Analysis, organization and querying of large genomic datasets using C++, Monsoon and various data structures.

  big-data-processingbioinformaticsdata-structures-and-algorithmsgenomic-sequences
  Language:C++ 2

• chuanting / imbalance_index

  全球电信资源分布不均衡指数刻画

  6gbig-data-processingdigital-divideconnect-the-unconnected
  Language:HTML 2

• IncredibleProgress / sweetheart.py

  rock-solid pillars for enterprise-grade solutions

  ubunturethinkdbrust-langtailwindcsspy-scriptjupyterbig-data-processingpythonrhelvuenginx-unit
  Language:Python 2

• JamesHanZhang / table-data-format-transform-app

  excel, markdown, csv, sql 数据源批量/单独格式互相转换

  big-data-processingcsv-to-excelcsv-to-sqldata-cleaning-pipelinedata-preprocessingeasy-to-useetl-frameworkexcel-to-mdmultifileupload
  Language:Python 2

• zaid-24 / Crack-Detection-using-CNN

  Crack Detection model using yolov7

  big-data-processingcnnpythonpytorchyolov7
  Language:Jupyter Notebook 2

• adnanrahin / NFL-Big-Data-Bowl-2022

  The 2022 Big Data Bowl data contains Next Gen Stats player tracking, play, game, player, and PFF scouting data for all 2018-2020 Special Teams play. Here, you'll find a summary of each data set in the 2022 Data Bowl, a list of key variables to join on, and a description of each variable.

  sparkscalabig-databig-data-processingrddspark-sql
  Language:Scala 1

• alessiococchieri / BDA-project-sparkify

  This Git repo showcases my analysis of Sparkify dataset with PySpark on Apache Spark cluster mode and JupyterLab on Docker. The goal was to identify at-risk customers and develop retention strategies. The analysis tested multiple machine learning models and uncovered insights into customer behavior and churn patterns.

  churn-analysischurn-predictionmachine-learningpysparksparksparkifyapache-sparkbig-databig-data-analyticsbig-data-processing
  Language:Jupyter Notebook 1

• Ayoub-etoullali / Activites-Pratiques-BigData

  MapReduce Job Development, RDDs Programming, Medical Data Management, Sales Analysis, And Efficient Data Integration For Big Data Analysis. Spark: Big Data Processing, SQOOP Integration, And Spark Structured Streaming For Real-Time Data.

  apache-sparkbig-data-analysisbig-data-processingdata-integrationmapreducemapreduce-javamapreduce-jobsmedical-data-managementpysparkrddsreal-timereal-time-datareal-time-databasesales-analysissparkspark-structured-streamingsqoopsqoop-exportsqoop-import
  Language:Java 1

• isandratskiy / awesome-bigdata-testing

  A list of awesome big data testing frameworks, resources and other awesomeness.

  testingawesomebig-datadataawesome-listbig-data-processingbig-data-testingautomationautomation-testingbig-data-automationdatabase
  1

• jamestiotio / dbsys

  SUTD 2021 50.043 Database and Big Data Systems Code Dump

  databasedatabase-managementdatabase-management-systemdatabase-systemsdatabase-designbig-databig-data-processingbig-data-analyticssqlnosqletlsparkpysparkhadoophdfsmapreduceb-plus-treezeppelindata-analysisdata-analytics
  Language:Java 1

• levindoneto / pandas-simple-csv-parser

  Simple CSV parser for huge volumes of data with the use of the library Pandas for Python for getting specific columns of a CSV file and putting the extracted data into one or more files (each column in a separated file or all of them in the same output) in a short amount of time.

  csvparserpandas-datareaderdata-manipulationconda-environmentbig-data-processingpandas-dataframes
  Language:Python 1

• louiecai / Sentiment-Analysis-API

  Sentiment-Analysis-API

  big-data-processingdeep-learninglstm-neural-networksmachine-learningneural-networknlprnn-pytorchsentiment-analysis
  Language:Python 1

• lucamoroz / BigDataComputing-UniPD

  Collection of homework (mostly Spark-based) from the course "Big Data Computing" - University of Padua.

  big-data-processingjavaspark
  Language:Java 1

• Mileristovski / DataEngineer-SparkStreaming

  Track a Boat est un système de suivi maritime en temps réel utilisant Kafka, Spark Structured Streaming et WebSockets. Il permet de visualiser la position des navires, analyser leurs trajectoires et prévoir leurs destinations sur une carte interactive.

  big-data-processingdistributed-computingdockerdocker-composekafkakafka-topicsmaritime-datapipelinepythonscalaspark-structured-streamingwebsocket
  Language:Jupyter Notebook 1

• RghdE / CapstoneTwo_EducationalLandscape

  Big Data and AI Engineering bootcamp 2nd capstone project. Using Big Data Tools to predict the probability of university enrollment for Egypt's High School students. :school: :books: :microscope:

  apache-pigbig-databig-data-analyticsbig-data-processingbig-data-projectsbig-data-visualizationdata-sciencemachine-learningpyspark
  Language:Jupyter Notebook 1