map-reduce

There are 5 repositories under map-reduce topic.

• chrislusf / gleam

  Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.

  distributed-computingdistributed-systemsgolangmap-reduce
  Language:Go 3408

• Qihoo360 / poseidon

  A search engine which can hold 100 trillion lines of log data.

  big-datagolangmap-reduceposeidonsearch-engine
  Language:Go 1963

• numaproj / numaflow

  Kubernetes-native platform to run massively parallel data/streaming jobs

  kubernetesstream-processingdata-processingk8spipelinemap-reducehacktoberfest
  Language:Go 1003

• JuliaFolds / Transducers.jl

  Efficient transducers for Julia

  juliatransducersparallelhigh-performancemap-reducedistributed-computingiterators
  Language:Julia 430
• Spark-with-Python

  tirthajyoti / Spark-with-Python

  Fundamentals of Spark with Python (using PySpark), code examples

  pysparksparkapache-sparkdataframemlibmachine-learningbig-datadatabasemap-reducepythonhdfsanalyticshadoopdistributed-computingparallel-computingsqlapache
  Language:Jupyter Notebook 324

• tkf / ThreadsX.jl

  Parallelized Base functions

  juliahigh-performancemap-reducetransducerssorting-algorithmsparallel
  Language:Julia 315

• commoncrawl / cc-mrjob

  Demonstration of using Python to process the Common Crawl dataset with the mrjob framework

  pythonmap-reducehadoopcommoncrawl
  Language:Python 166

• phelps-sg / python-bigdata

  Data science and Big Data with Python

  data-sciencepythonhbasenumpynumerical-methodsnotebook-jupytersparkmap-reduce
  Language:Jupyter Notebook 130

• xarray-contrib / flox

  Fast & furious GroupBy operations for dask.array

  daskmap-reducexarray
  Language:Python 120

• asavinov / prosto

  Prosto is a data processing toolkit radically changing how data is processed by heavily relying on functions and operations with functions - an alternative to map-reduce and join-groupby

  workflowdata-processingmap-reducesparkpandaspythonfeature-engineeringdata-sciencedata-wranglingdata-preprocessingdata-preparationbusiness-intelligenceolap
  Language:Python 90

• daleroberts / pypar

  Efficient and scalable parallelism using the message passing interface (MPI) to handle big data and highly computational problems.

  map-reducempipythonbig-data
  Language:Python 69

• JuliaFolds / FoldsCUDA.jl

  Data-parallelism on CUDA using Transducers.jl and for loops (FLoops.jl)

  gpucudajuliatransducersparallelhigh-performancemap-reduceiterators
  Language:Julia 56

• rvantonder / hack_parallel

  The core parallel and shared memory library used by Hack, Flow, and Pyre

  ocamlparallelshared-memorymap-reduce
  Language:OCaml 42

• dragonly / pingcap_interview

  pingcap 面试小作业

  interviewmap-reduce
  Language:Go 36

• JuliaFolds / data-parallelism

  juliaparallelhigh-performancemap-reducedistributed-computingiteratorstransducersfranklin
  Language:Julia 36

• RedisGears / redisgears-py

  RedisGears python client

  redisredisgearspython-clientmap-reducestream-processing
  Language:Python 27

• Cheng-Lin-Li / Spark

  There are Python 2.7 codes and learning notes for Spark 2.1.1

  python27kmeanskmeans-clusteringalsapriori-algorithmminhash-lsh-algorithmminhashuv-decompositionalternating-least-squaressavasere-omiecinski-and-navatheapriori-sonsparkmap-reducetf-idfcosine-similarity
  Language:Python 24

• CaptainCodeman / datastore-mapper

  Appengine Datastore Mapper in Go

  datastore-mapperdatastore-entitiescloud-storageshardsbigquerydatastoreappenginegomap-reduce
  Language:Go 21

• nglthu / infoRetrieval

  Inverted Indexer, web crawler, sort, search and poster steamer written using Python for information retrieval.

  python3inverted-indexmap-reducetermstokensheapswebcrawlerstemming-algorithminformation-retrieval
  Language:HTML 20

• ihor / Phadoop

  Map/reduce jobs for Hadoop in PHP

  phphadoopmap-reduce
  Language:PHP 18

• loveyacper / raft_for_dummies

  2017春季MIT分布式系统课程实验

  consensus-algorithmgolangmap-reduceraft
  Language:Go 18
• open-soql

  shellyln / open-soql

  Open source implementation of the SOQL.

  soqlgraph-queryobject-querysqlresolversdmlmap-reducejavascripttypescriptlibraryquery-engine
  Language:TypeScript 16

• imehrdadmahdavi / map-reduce-inverted-index

  Creating an Inverted Index of words occurring in a large set of documents extracted from web pages using Hadoop MapReduce and Google Dataproc

  mapreducehadoopinverted-indexmap-reduceinformation-retrievalgcpdataprocclusteringbigdatabig-datagooglecloudsearch-enginedataprocessing
  Language:Java 14

• pscosta / go-strm

  A rich Map/Reduce API in Go

  mapreducegenericsgolangfunctional-programmingmap-reduce
  Language:Go 14
• jsdp

  gwr3n / jsdp

  A Java Stochastic Dynamic Programming Library

  stochasticdynamicprogrammingjavauncertaintyobject-orientedparallelmap-reducelambda-calculusstreaminventoryoptimalcontrolmaintenance
  Language:Java 13

• HenrikBengtsson / future.mapreduce

  [EXPERIMENTAL] R package: future.mapreduce - Utility Functions for Future Map-Reduce API Packages

  rpackagefuturesmap-reduce
  Language:R 13

• kalmyk / fox-wamp

  Web Application Message Async Server and WAMP/MQTT bridge

  mqttwebsocketiotmap-reducestream-processingwamp-routerasync-storage
  Language:JavaScript 13

• fangvv / EdgeLD

  Code for paper "Locally Distributed Deep Learning Inference on Edge Device Clusters"

  dnninferenceclusterparallel-computingspeedupdeep-learningedge-computingvggnetdistributed-computingmap-reduceparallel-algorithmworkload
  Language:Python 12

• lorenzo-stacchio / Big_Data_Course_Rimini_2021

  Questa repository contiene tutto il materiale didattico utilizzato durante il corso di "Laboratorio Big Data" in collaborazione con il comune di Rimini.

  data-sciencebig-datanosql-databasesql-databasemachine-learningdata-visualizationregressionclassificationfeature-extractionfeature-selectionsparkhadoopmap-reduce
  Language:Jupyter Notebook 11

• codelibra / Time-series-analysis-nyc-taxi

  ⏰ 📓 Time series analysis of new york taxi data

  map-reducehadoophivetime-seriestaxi-datanew-york-citymachine-learningnightlife
  Language:Java 9

• deeperunderstanding / scala-map-reduce

  Map-Reduce implemented with Scala

  actorfuturemap-reduceparallelscala
  Language:Scala 8

• DICL / VeloxMR

  Data processing component of the Velox Big Data Framework (VBDF)

  map-reducedfshadoopsparkiterationhpcbig-data
  Language:C++ 8

• pipecraft / pipes

  Core part of pipes framework plus some commonly used extensions

  data-processingdistributed-computingmap-reducepipelines
  Language:Java 8

• 123vivekr / distributed-map-reduce

  An experimental distributed map reduce system based on Google's MapReduce, written in Rust!

  distributed-systemsmap-reducerust-lang
  Language:Rust 7

• asuiu / streamerate

  Iterable Java8 style Streams for Python

  java-streamsmap-reducemapreducepythonpython-iterablespython-itertoolspython-mapreducepython-multiprocessingpython-multithreadingpython-streamingpython3streaming
  Language:Python 7

• natelalor / AI_report_generator

  A tool that converts long audio files into a thorough, summarized report. Leverages OpenAI and its API (ChatGPT backend), Langchain for text processing, and Pinecone for vector database facilitation.

  artificial-intelligencechatbotembedding-modelslangchainmap-reduceobject-oriented-programmingopenaiopenai-apipineconepythonvector-database
  Language:Python 7