hadoop-filesystem

There are 7 repositories under hadoop-filesystem topic.

• lakeFS

  treeverse / lakeFS

  lakeFS - Data version control for your data lake | Git for data

  apache-sparkapache-sparksqlaws-s3azure-blob-storageazure-storagedata-engineeringdata-lakedata-qualitydata-version-controldata-versioningdatalakedatalakesgit-for-datagogolanggoogle-cloud-storagehadoop-filesystemlakefsobject-storage
  Language:Go 4965

• GoogleCloudDataproc / hadoop-connectors

  Libraries and tools for interoperability between Hadoop-related open-source software and Google Cloud Platform.

  bigquerygoogle-cloud-dataprochadoophadoop-filesystemhadoop-hcfs
  Language:Java 286

• linkedin / dynamometer

  A tool for scale and performance testing of HDFS with a specific focus on the NameNode.

  hadoophadoop-filesystemhadoop-frameworkhadoop-hdfshdfshdfs-dfsperformance-analysisperformance-metricsperformance-testperformance-testingscalescale-uptestingtesting-tools
  Language:Java 134
• Data-Engineering-Project-with-HDFS-and-Kafka

  AhmetFurkanDEMIR / Data-Engineering-Project-with-HDFS-and-Kafka

  Data Engineering Project with Hadoop HDFS and Kafka

  datadata-engineerdata-engineeringdata-engineering-pipelinedockerdocker-composehadoophadoop-filesystemhadoop-hdfshdfshdfs-clienthdfs-dfskafkakafka-consumerkafka-producerkafka-uipiplinepythonpython-hdfs-clientkafkaui
  Language:Python 118

• mmolimar / kafka-connect-fs

  Kafka Connect FileSystem Connector

  apache-kafkaazure-storageconfluentfilesfilesystemftpgcphadoophadoop-filesystemhdfskafkakafka-connectkafka-connect-fskafka-connectors3
  Language:Java 112

• jingw / pyhdfs

  Python HDFS client

  hadoophadoop-filesystemhdfspythonwebhdfs
  Language:Python 94

• longshilin / HDFS-Netdisc

  基于Hadoop的分布式云存储系统 :palm_tree:

  bigdatafilesystemhadoophadoop-filesystemhdfshdfs-clienthdfs-netdiscnetdisk
  Language:Java 78

• vivek2319 / Learn-Hadoop-and-Spark

  This repository focuses on gathering and making a curated list resources to learn Hadoop for FREE.

  hadoophadoop-filesystembigdatamapreducepigooziehivehbaseflumesqoopyarnhiveqlimpalaapache-kafkaapache-sparkapache-stormapache-solrapache-atlasapache-rangerapache-knox
  Language:Python 55

• palantir / hadoop-crypto

  Library for per-file client-side encyption in Hadoop FileSystems such as HDFS or S3.

  hadoop-filesystemhadoop-cryptohadoopocto-correct-managed
  Language:Java 48

• pfisterer / apache-hadoop-helm

  Helm chart for Apache Hadoop using multi-arch docker images

  dockerhadoophadoop-filesystemhadoop-hdfshadoop-mapreducehelmhelm-chartkubernetes
  Language:Dockerfile 20
• datasqueeze

  ExpediaGroup / datasqueeze

  Hadoop utility to compact small files

  hadoophadoop-filesystemhadoop-smallfilessmallfileshdfshdfs-compaction
  Language:Java 18

• averyzhong / hdfs-over-sftp

  SFTP server which works on the top of HDFS，It is based on Apache sshd to access and operate HDFS through SFTP protocol

  hdfssftphdfs-over-sftphadoop-filesystem
  Language:Java 12

• waltherg / distributable_docker_sql_on_hadoop

  Toy Hadoop cluster combining various SQL-on-Hadoop variants

  hadoophadoop-mapreducehadoop-filesystemhadoop-clusterhadoop-dockerhadoop-hdfshadoop-frameworkhivehuesparksparksqlhbasehbase-clientyarnyarn-hadoop-clusterzookeeperzookeeper-deploymenttezimpalapresto
  Language:Shell 12

• procter-gamble-oss / octopufs

  OctopuFS library helps managing cloud storage, ADLSgen2 specifically. It allows you to operate on files (moving, copying, setting ACLs) in very efficient manner. Designed to work on databricks, but should work on any other platform as well.

  adlsgen2azure-storagedatabrickshadoop-filesystemscalaspark
  Language:Scala 11

• Tapad / sbt-hadoop-oss

  An sbt plugin for publishing artifacts to HDFS.

  hadoophadoop-filesystemhdfssbt
  Language:Scala 10

• fasouto / webhdfspy

  Python wrapper to access Hadoop HDFS REST API

  hadoop-filesystemhdfspythonwrapper
  Language:Python 8

• christopherkindl / twitter-data-pipeline-using-airflow-and-apache-spark

  Data pipeline to process and analyse Twitter data in a distributed fashion using Apache Spark and Airflow in AWS environment

  python3apache-sparkawsairflowhadoop-filesystem
  Language:Python 7

• jazzwang / hadoop_labs

  MapReduce Java Code Examples to learn Hadoop

  hadoop-filesystemhadoop-mapreducejava
  Language:Java 7

• TritonDataCenter / hadoop-manta

  Hadoop Filesystem Driver for Manta

  drillhadoophadoop-filesystemjoyentmantasqooptriton
  Language:Java 6

• aadishgoel / Hadoop-Codes

  Neat and Handy Place for all Hadoop codes

  hadoopmapreduce-javajavaapihdfswordcounthadoop-hdfshadoop-mapreducehadoop-filesystem
  Language:Java 5

• HxnDev / Finding-Average-Temperature-of-Each-Year-using-Hadoop-HDFS

  In this task, we had to calculate the average temperature for each year from the given dataset using Hadoop HDFS. We had to create a MapReduce function to perform this task.

  hadoophadoop-mapreducehadoop-hdfshadoop-filesystemhadoop-clustermapreducemapreduce-javaaverage-calculatorcodejava
  Language:Java 5

• CUBigDataClass / soccer-tweet-analysis

  Ingestion pipeline to analyze soccer tweets

  pythoncassandrahadoop-filesystemtravis-cisparkelasticsearchangularjsdjangoredis
  Language:Python 4

• HxnDev / Hadoop-MapReduce-to-Find-Average-Length-of-Comments

  In this task, we had to find the average length of comments given in the dataset. It was done using Hadoop MapReduce and Hadoop HDFS.

  average-lengthcodecommentsdistributed-computingdistributed-systemshadoophadoop-filesystemhadoop-hdfshadoop-mapreducehdfsjavamapreducemapreduce-javaparallel-computingparallel-programming
  Language:Java 4

• Mohammed-siddiq / hadoop-XMLInputFormatWithMultipleTags

  Mahout's XMLInputFormat with support for multiple input and output tags.

  hadoop-filesystemxmlinputformatmahout
  Language:Java 4

• swan815 / MyFirstHadoopYunpan

  基于hadoop的简易云盘实现

  javawebjsphadoop-filesystem
  Language:CSS 4

• tchaye59 / Hadoop-Perfect-File

  A Fast access container for small files

  hadoophadoop-filesystem
  Language:Java 4

• rshad / OpenCCML

  Category: Cloud Computing and Machine Learning Application - Subject: A cloud platform to make data processing with machine learning algorithms, built on Openstack, using Spark for data distribution and Hadoop Filesystem for data storage

  ansiblecloud-computingdata-processinghadoop-filesystemmachine-learningnovaopenstackspark
  Language:Python 3

• SarahAyaz / YouTube_Data_Analysis

  Analysis of YouTube Data using Hadoop Mapreduce framework in Java.

  analysishadoophadoop-ecosystemhadoop-filesystemhadoop-hdfshadoop-mapreducehdfsjavalinuxmapreducemapreduce-javapartitioneryoutube
  Language:Java 3

• alex-ber / docker-hive

  EMR 5.25.0 cluster single node Hadoop docker image. With Amazon Linux, Hadoop 2.8.5 and Hive 2.3.5

  hadoop-dockerhivedockerdocker-composedockerfiledocker-imagehadoop-hdfshadoop-mapreducehadoop-clusterhadoop-ecosystemhadoop-frameworkhadoop-filesystemyarn-hadoop-clusteryarnhiveserver2dockerfilesdocker-imageshadoopemremr-cluster
  Language:Shell 2

• huangyueranbbc / hadoop05_pagerank

  pagerank hadoop

  hadoophadoop-mapreducemapreducemapreduce-jobspagerankpagerank-mapreducehadoop-clusterhadoop-filesystemhadoop-javajavamapreduce-javapagerank-javapagerank-algorithmjava-pagerank
  Language:Java 2

• humanbeeng / hadoop-auto-install

  A small helper script that can save your valuable time during installation of Apache Hadoop.

  hadoophadoop-hdfshadoop-filesystemhadoop-clusterbash-script
  Language:Shell 2

• HwaiTengTeoh / Airbnb-Big-Data-Management

  To develop an Airbnb database and create a pipeline using MongoDB and Hadoop architecture to ease the process of managing, loading, processing, querying, and analyzing Airbnb data based on location

  big-data-managementhdfshiveqlpig-latinpyspark-pythonhadoop-filesystem
  Language:Jupyter Notebook 2

• mikeroyal / Apache-Hadoop-Guide

  Apache Hadoop Guide

  hadoophadoop-mapreducehadoop-hdfshadoop-clusterhadoop-filesystem
  2

• Niranjankumar-c / DataAnalytics_using_ClickstreamData

  Casestudy completed as part of BigData training from analytix labs

  hivehadoop-filesystemexcelsqoop
  2

• samarthtambad / big-data-pl

  Analysing programming languages by community characteristics on Github and StackOverflow

  analyticsbig-datahadoop-filesystemhdfssbtscalasparktableau-desktop
  Language:Scala 2

• Rohit9314 / my-hadoop

  Setup hadoop cluster manually and automatically

  hadoop-dockerhadoop-clusterhadoop-mapreducehadoop-filesystemhadoop-frameworkhadoop-distributionshdfs-dockerdocker-containerdockerfilesdocker-implemented-hadoopautomated-hadoop-implementationcomplete-hadoop-setuphadoop-using-devops
  Language:Python 1