hadoop-hdfs

There are 14 repositories under hadoop-hdfs topic.

• seaweedfs

  seaweedfs / seaweedfs

  SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.

  distributed-storagedistributed-systemss3hdfsfusedistributed-file-systemhadoop-hdfsposixtiered-file-systemkubernetesreplicationobject-storages3-storageseaweedfserasure-codingblob-storagecloud-drive
  Language:Go 21056

• OBenner / data-engineering-interview-questions

  More than 2000+ Data engineer interview questions.

  data-engineeringinterview-questionsinterviewhadoophadoop-hdfssparkflinksqlkafkahiveimpalaairflowawsazurecassandraflumehbaseavronifidata-structures
  785

• Morphl-AI / MorphL-Community-Edition

  MorphL Community Edition uses big data and machine learning to predict user behaviors in digital products and services with the end goal of increasing KPIs (click-through rates, conversion rates, etc.) through personalization

  artificial-intelligencecassandraconversion-rate-optimizationdata-driven-designfront-end-developmenthadoop-hdfskubernetesmachine-learningmorphl-platformpipelineproduct-developmentpysparkuser-experience
  Language:Python 259

• linkedin / dynamometer

  A tool for scale and performance testing of HDFS with a specific focus on the NameNode.

  hadoophadoop-filesystemhdfshdfs-dfstestingtesting-toolsscalescale-upperformance-testingperformance-testperformance-analysisperformance-metricshadoop-frameworkhadoop-hdfs
  Language:Java 129

• IBM / sparksql-for-hbase

  Learn how to use Spark SQL and HSpark connector package to create / query data tables that reside in HBase region servers

  hbasesparksqlnosqlhadoop-hdfsapache-sparkibmcode
  68
• big_data

  groda / big_data

  Tutorials on Big Data essentials: Hadoop, MapReduce, Spark.

  apache-sedonabig-databigdatadockergutenberg-ebookshadoophadoop-clusterhadoop-hdfsjupyter-notebookmapreducemapreduce-bashmrjobpysparksparkspark-sqltestdfsio
  Language:Jupyter Notebook 59

• vim89 / datapipelines-essentials-python

  Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

  apache-sparkbig-datadata-pipelinedatalakeetletl-componentsetl-frameworketl-pipelinehadoophadoop-hdfshadoop-mapreducepysparkpythonpython3sparkspark-sqlxmlxml-parsing
  Language:Python 53

• maniram-yadav / Big_DataHadoop_Projects

  Big data projects implemented by Maniram yadav

  big-data-analyticsbig-data-projectsflumehadoophadoop-hdfshadoop-mapreducehdfshivemapreducepigpig-latinsparksqoop
  Language:PigLatin 48

• hokstack / hok-helm

  HokStack - Run Hadoop Stack on Kubernetes

  automationbigdatadataopsdevops-toolshadoophadoop-clusterhadoop-hdfshdpkubernetesoperator
  Language:Shell 22

• hundredlabs / console

  Open source data infrastructure platform. Designed for developers, built for speed.

  sparkkafkahadoophadoop-hdfsdata-ops
  Language:TypeScript 22
• TravelWebsite_BigDataAnalysis

  jarlor / TravelWebsite_BigDataAnalysis

  旅游网站(携程网部分数据)大数据分析-hadoop课程设计(本科课设级别)

  bigdatacourseworkhadoop-hdfsjavamapreduce
  Language:Java 18

• hadoop-sandbox / hadoop-sandbox

  A fully-functional Hadoop Yarn cluster as docker-compose deployment.

  dockerdocker-composehadoophadoop-yarnhadoop-hdfshadoop-cluster
  Language:Shell 15

• lucas91batista / twitter-hashtag-graph

  Twitter + Flume + Hadoop (HDFS, MapReduce) + Neo4j + Pyhton

  twitterapache-flumehadoophadoop-mapreducehadoop-hdfsneo4j
  Language:JavaScript 15

• SepehrImanian / ansible-hadoop-hdfs

  Ansible Playbook For Setup Hadoop HDFS

  ansibleansible-playbookhadoophadoop-hdfshdfs
  Language:Jinja 14

• PChou / marayarn

  Marathon on yarn

  hadoop-hdfsmarathonyarn
  Language:Java 13

• pfisterer / apache-hadoop-helm

  Helm chart for Apache Hadoop using multi-arch docker images

  hadoophadoop-hdfshadoop-mapreducehadoop-filesystemhelm-charthelmdockerkubernetes
  Language:Dockerfile 13

• alagrede / HdfsClient

  A Java Hdfs client example and full Kerberos example for call hadoop commands directly in java code or on your local machine.

  hadoopkerberoskerberos-authenticationhadoop-hdfs
  Language:Java 12

• waltherg / distributable_docker_sql_on_hadoop

  Toy Hadoop cluster combining various SQL-on-Hadoop variants

  hadoophadoop-mapreducehadoop-filesystemhadoop-clusterhadoop-dockerhadoop-hdfshadoop-frameworkhivehuesparksparksqlhbasehbase-clientyarnyarn-hadoop-clusterzookeeperzookeeper-deploymenttezimpalapresto
  Language:Shell 12

• Areesha-Tahir / Hadoop-MapReduce-Sentiment-Analysis-Through-Keywords

  A MapReduce program to conduct sentiment analysis of a keyword from a list of comments.

  mapreduceparallel-computingparallel-programmingsentiment-analysiscodeprojectjavaubuntuhadoop-mapreducehadoop-hdfshadoop
  Language:Java 10
• Data-Engineering-Project-with-HDFS-and-Kafka

  AhmetFurkanDEMIR / Data-Engineering-Project-with-HDFS-and-Kafka

  Data Engineering Project with Hadoop HDFS and Kafka

  datadata-engineerdata-engineeringdata-engineering-pipelinedockerdocker-composehadoophadoop-filesystemhadoop-hdfshdfshdfs-clienthdfs-dfskafkakafka-consumerkafka-producerkafka-uipiplinepythonpython-hdfs-clientkafkaui
  Language:Python 8

• jodth07 / hadoop-installation

  Instructions on setting up Hadoop, HDFS, java, sbt, kafka, scala, spark and flume on Ubuntu 18.04

  flumehadoophadoop-ecosystemhadoop-hdfshadoop-installationinstallationkafkakafka-installationsbtsbt-installationscalascala-installationsparkspark-installation
  Language:Shell 8

• leibniz21c / mammoth

  Mammoth is a container based hadoop distributed system log analyzer. Sponsed by Mantech and Naver Cloud Platform.

  msalog-analyzerinfluxdbmongodbadobe-xdfastapihadoop-hdfsyarnmapreduceflutter-appdartpython3dockerdocker-compose
  Language:Dart 8

• LMAPcoder / Hadoop-on-Colab

  Installation and configuration of Hadoop on Google Colaboratory

  hadoophadoop-hdfshadoop-installationhadoop-mapreducehadoop-streaming
  Language:Jupyter Notebook 7

• mgarralda / hadoop-spark-cluster

  Repository containing Docker images for create a cluster Spark on Hadoop Yarn.

  hadoop-hdfsspark-clusterspark-hadoopspark-hadoop-dockerspark-yarn-dockerspark
  Language:Dockerfile 7

• aadishgoel / Hadoop-Codes

  Neat and Handy Place for all Hadoop codes

  hadoopmapreduce-javajavaapihdfswordcounthadoop-hdfshadoop-mapreducehadoop-filesystem
  Language:Java 6

• HxnDev / Hadoop-MapReduce-to-Analyze-Sentiment-of-Keyword

  In this task, we had to write a MapReduce program to analyze the sentiment of a keyword from a list of comments. This was done using Hadoop HDFS.

  hadoophadoop-mapreducehadoop-hdfsmapreducemapreduce-javahdfsparallel-computingparallel-programmingsentiment-analysissentimental-analysisjavacode
  Language:Java 6

• prabal03 / python-automation-in-linux

  Python automation in linux

  hadoop-hdfsdockerwebserverlvmawslinux
  Language:Python 6

• berksudan / Distributed-Environment-Installation-Guide

  Install Hadoop, HDFS, Yarn and Spark on 3 Ubuntu 18.04 Machines

  hadoophadoop-hdfssparkinstallation-guidesvirtual-machines
  5

• HxnDev / Finding-Average-Temperature-of-Each-Year-using-Hadoop-HDFS

  In this task, we had to calculate the average temperature for each year from the given dataset using Hadoop HDFS. We had to create a MapReduce function to perform this task.

  hadoophadoop-mapreducehadoop-hdfshadoop-filesystemhadoop-clustermapreducemapreduce-javaaverage-calculatorcodejava
  Language:Java 5

• Reza-Marzban / Vehicle-Fuel-Hadoop-MapReduce

  Vehicle Fuel Hadoop MapReduce

  hadoopmapreducemapreduce-javabigdatahadoop-mapreducehadoop-hdfs
  Language:Java 5

• HxnDev / Hadoop-MapReduce-to-Find-Average-Length-of-Comments

  In this task, we had to find the average length of comments given in the dataset. It was done using Hadoop MapReduce and Hadoop HDFS.

  hadoophadoop-mapreducehadoop-hdfshdfshadoop-filesystemaverage-lengthcommentsparallel-computingparallel-programmingdistributed-systemsdistributed-computingcodejavamapreducemapreduce-java
  Language:Java 4

• jomarsilio / Bootcamp-IGTI-Analista-de-Dados

  Bootcamp ministrado pela IGTI com o objetivo de abordar de forma intensiva conceitos e práticas da análise de dados, habilitando o aluno para atuar profissionalmente na área.

  anacondagoogle-colabhadoop-hdfshadoop-mapreducehadoop-streamingjupyter-notebookkmeans-clusteringknimenosqlphytonpowerbiprotegersql
  Language:Jupyter Notebook 4

• karamolegkos / Twitter_Data_Analyzer

  avrobig-datacssflafkaflumehadoophadoop-hdfshtmlinformation-systems-governancejavajavascriptjspkafkamahoutmap-reducetwiitertwitter-api
  Language:Java 4

• MarceloJSSantos / acelereracao-global-dev-4-everis-dio

  Repositório criado para armazenar anotações e atividades desempenhadas no treinamento na plataforma da Digital Inovattion One (DIO) para o Processo seletivo de Engenheiros de Dados pela empresa Everis.

  data-engineeringsqoopbig-datahadoop-hdfsyarnhivenosqlhbasecassandrapythonkafkasparkscalapysparkdiolinux
  Language:Jupyter Notebook 4

• prithvianilk / rdfs

  An attempt to make a reliable, distributed file system inspired by HDFS

  distributed-file-systemhadoop-hdfs
  Language:Java 4

• waikeungt / hdfs-spring-boot-starter

  用于spring boot快捷使用HDFS的starter

  gradlehadoophadoop-hdfshdfsjavaspring-boot
  Language:Java 4