hadoop-hdfs

There are 14 repositories under hadoop-hdfs topic.

• seaweedfs

  seaweedfs / seaweedfs

  SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, xDC replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding. Enterprise version is at seaweedfs.com.

  distributed-storagedistributed-systemss3hdfsfusedistributed-file-systemhadoop-hdfsposixtiered-file-systemkubernetesreplicationobject-storages3-storageseaweedfserasure-codingblob-storagecloud-drive
  Language:Go 25793

• OBenner / data-engineering-interview-questions

  More than 2000+ Data engineer interview questions.

  data-engineeringinterview-questionsinterviewhadoophadoop-hdfssparkflinksqlkafkahiveimpalaairflowawsazurecassandraflumehbaseavronifidata-structures
  1408

• Morphl-AI / MorphL-Community-Edition

  MorphL Community Edition uses big data and machine learning to predict user behaviors in digital products and services with the end goal of increasing KPIs (click-through rates, conversion rates, etc.) through personalization

  artificial-intelligencemachine-learninguser-experienceconversion-rate-optimizationfront-end-developmentdata-driven-designproduct-developmentmorphl-platformpysparkcassandrakuberneteshadoop-hdfspipeline
  Language:Python 261

• linkedin / dynamometer

  A tool for scale and performance testing of HDFS with a specific focus on the NameNode.

  hadoophadoop-filesystemhdfshdfs-dfstestingtesting-toolsscalescale-upperformance-testingperformance-testperformance-analysisperformance-metricshadoop-frameworkhadoop-hdfs
  Language:Java 131
• Data-Engineering-Project-with-HDFS-and-Kafka

  AhmetFurkanDEMIR / Data-Engineering-Project-with-HDFS-and-Kafka

  Data Engineering Project with Hadoop HDFS and Kafka

  datadata-engineerdata-engineeringdata-engineering-pipelinedockerdocker-composehadoophadoop-filesystemhadoop-hdfshdfshdfs-clienthdfs-dfskafkakafka-consumerkafka-producerkafka-uipiplinepythonpython-hdfs-clientkafkaui
  Language:Python 117
• big_data

  groda / big_data

  Tutorials on Big Data essentials: Hadoop, MapReduce, Spark. Explore a variety of tutorials and demonstrations on Big Data technologies, primarily in the form of Jupyter notebooks. Most notebooks are self-contained and live—ready to run with a click.

  big-databigdatasparkspark-sqldockermapreducemapreduce-bashpysparkhadooptestdfsiojupyter-notebookapache-sedonahadoop-clusterhadoop-hdfsmrjobgutenberg-ebookshadoop-mapreduceapache-sparkbigtop
  Language:Jupyter Notebook 73

• IBM / sparksql-for-hbase

  Learn how to use Spark SQL and HSpark connector package to create / query data tables that reside in HBase region servers

  hbasesparksqlnosqlhadoop-hdfsapache-sparkibmcode
  69

• vim89 / datapipelines-essentials-python

  Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

  apache-sparksparkspark-sqlpythonpython3pysparketletl-pipelineetl-frameworketl-componentsxmlxml-parsingdatalakebig-datahadoophadoop-mapreducehadoop-hdfsdata-pipeline
  Language:Python 55

• maniram-yadav / Big_DataHadoop_Projects

  Big data projects implemented by Maniram yadav

  sparkpig-latinpighadoophdfssqoophivemapreducebig-data-analyticsbig-data-projectshadoop-mapreducehadoop-hdfsflume
  Language:PigLatin 51
• TravelWebsite_BigDataAnalysis

  jarlor / TravelWebsite_BigDataAnalysis

  旅游网站(携程网部分数据)大数据分析-hadoop课程设计(本科课设级别)

  bigdatacourseworkhadoop-hdfsjavamapreduce
  Language:Java 37

• hokstack / hok-helm

  HokStack - Run Hadoop Stack on Kubernetes

  hadoophdpkubernetesoperatorbigdatahadoop-clusterhadoop-hdfsautomationdataopsdevops-tools
  Language:Shell 24

• torqbit / databox

  Open source data infrastructure platform. Designed for developers, built for speed.

  data-opshadoophadoop-hdfskafkaspark
  Language:TypeScript 22

• SepehrImanian / ansible-hadoop-hdfs

  Ansible Playbook For Setup Hadoop HDFS

  ansibleansible-playbookhadoophadoop-hdfshdfs
  Language:Jinja 20

• hadoop-sandbox / hadoop-sandbox

  A fully-functional Hadoop Yarn cluster as docker-compose deployment.

  dockerdocker-composehadoophadoop-clusterhadoop-hdfshadoop-yarn
  Language:Shell 18

• pfisterer / apache-hadoop-helm

  Helm chart for Apache Hadoop using multi-arch docker images

  hadoophadoop-hdfshadoop-mapreducehadoop-filesystemhelm-charthelmdockerkubernetes
  Language:Dockerfile 18

• lucas91batista / twitter-hashtag-graph

  Twitter + Flume + Hadoop (HDFS, MapReduce) + Neo4j + Pyhton

  twitterapache-flumehadoophadoop-mapreducehadoop-hdfsneo4j
  Language:JavaScript 15

• PChou / marayarn

  Marathon on yarn

  marathonyarnhadoop-hdfs
  Language:Java 13

• alagrede / HdfsClient

  A Java Hdfs client example and full Kerberos example for call hadoop commands directly in java code or on your local machine.

  hadoophadoop-hdfskerberoskerberos-authentication
  Language:Java 12

• waltherg / distributable_docker_sql_on_hadoop

  Toy Hadoop cluster combining various SQL-on-Hadoop variants

  hadoophadoop-mapreducehadoop-filesystemhadoop-clusterhadoop-dockerhadoop-hdfshadoop-frameworkhivehuesparksparksqlhbasehbase-clientyarnyarn-hadoop-clusterzookeeperzookeeper-deploymenttezimpalapresto
  Language:Shell 12

• Areesha-Tahir / Hadoop-MapReduce-Sentiment-Analysis-Through-Keywords

  A MapReduce program to conduct sentiment analysis of a keyword from a list of comments.

  mapreduceparallel-computingparallel-programmingsentiment-analysiscodeprojectjavaubuntuhadoop-mapreducehadoop-hdfshadoop
  Language:Java 11

• jodth07 / hadoop-installation

  Instructions on setting up Hadoop, HDFS, java, sbt, kafka, scala, spark and flume on Ubuntu 18.04

  hadoophadoop-hdfshadoop-ecosystemkafkainstallationscalasbtsparkflumespark-installationhadoop-installationkafka-installationsbt-installationscala-installation
  Language:Shell 8

• leibniz21c / mammoth

  Mammoth is a container based hadoop distributed system log analyzer. Sponsed by Mantech and Naver Cloud Platform.

  msalog-analyzerinfluxdbmongodbadobe-xdfastapihadoop-hdfsyarnmapreduceflutter-appdartpython3dockerdocker-compose
  Language:Dart 8

• Mahmoud-nfz / football-big-data

  This is a comprehensive solution for real-time football analytics, leveraging Apache Spark execution on yarn for both streaming and batch processing, Hadoop HDFS for distributed storage, Kafka for real-time data ingestion, rethinkdb for live data updates , a custom built search engine and Next.js for data visualization.

  hadoophadoop-hdfskafkanextjssearch-enginesparkspark-streamingt3-stackrethinkdb
  Language:TypeScript 8

• LMAPcoder / Hadoop-on-Colab

  Installation and configuration of Hadoop on Google Colaboratory

  hadoophadoop-hdfshadoop-installationhadoop-mapreducehadoop-streaming
  Language:Jupyter Notebook 7

• mgarralda / hadoop-spark-cluster

  Repository containing Docker images for create a cluster Spark on Hadoop Yarn.

  hadoop-hdfsspark-clusterspark-hadoopspark-hadoop-dockerspark-yarn-dockerspark
  Language:Dockerfile 7

• briandi26 / Machine-Learning-for-Forest-Fire-Prediction

  Machine Learning for Forest Fire Prediction using Hadoop ecosystems and Spark Tools (Pyspark)

  machine-learningforest-fire-modelhadoop-hdfssparkpyspark
  Language:Python 6

• HxnDev / Hadoop-MapReduce-to-Analyze-Sentiment-of-Keyword

  In this task, we had to write a MapReduce program to analyze the sentiment of a keyword from a list of comments. This was done using Hadoop HDFS.

  hadoophadoop-mapreducehadoop-hdfsmapreducemapreduce-javahdfsparallel-computingparallel-programmingsentiment-analysissentimental-analysisjavacode
  Language:Java 6

• prabal03 / python-automation-in-linux

  Python automation in linux

  hadoop-hdfsdockerwebserverlvmawslinux
  Language:Python 6

• Ren294 / Covid-Data-Process

  This project integrates real-time data processing and analytics using Apache NiFi, Kafka, Spark, Hive, and AWS services for comprehensive COVID-19 data insights.

  airflowawsaws-ec2aws-quicksightbig-datacovid19-datadockerdocker-composehadoop-hdfshdfshivekafkanifiredpandasparkspark-sqlspark-streamingbig-data-analyticspipelinesparksql
  Language:Shell 6

• aadishgoel / Hadoop-Codes

  Neat and Handy Place for all Hadoop codes

  hadoopmapreduce-javajavaapihdfswordcounthadoop-hdfshadoop-mapreducehadoop-filesystem
  Language:Java 5

• berksudan / Distributed-Environment-Installation-Guide

  Install Hadoop, HDFS, Yarn and Spark on 3 Ubuntu 18.04 Machines

  hadoophadoop-hdfsinstallation-guidessparkvirtual-machines
  5

• hadoop-sandbox / hadoop-sandbox-images

  Docker image builds for Hadoop sandbox.

  dockerhadoophadoop-clusterhadoop-hdfshadoop-yarnhdfs
  Language:Dockerfile 5

• HxnDev / Finding-Average-Temperature-of-Each-Year-using-Hadoop-HDFS

  In this task, we had to calculate the average temperature for each year from the given dataset using Hadoop HDFS. We had to create a MapReduce function to perform this task.

  hadoophadoop-mapreducehadoop-hdfshadoop-filesystemhadoop-clustermapreducemapreduce-javaaverage-calculatorcodejava
  Language:Java 5

• Ren294 / Log-Analysis-Project

  This project builds a scalable log analytics pipeline use Lambda architecture for real-time and batch processing of NASA server logs.

  big-datadata-engineeringdata-scienceapache-kafkaapache-nifiapache-sparkcassandracassandra-drivergrafanahadoophadoop-hdfshivepowerbispark-rddspark-sqlspark-streamingbig-data-analytics
  Language:Python 5

• Reza-Marzban / Vehicle-Fuel-Hadoop-MapReduce

  Vehicle Fuel Hadoop MapReduce

  hadoopmapreducemapreduce-javabigdatahadoop-mapreducehadoop-hdfs
  Language:Java 5

• waikeungt / hdfs-spring-boot-starter

  用于spring boot快捷使用HDFS的starter

  hdfsspring-bootgradlejavahadoophadoop-hdfs
  Language:Java 4