parquet-files

There are 1 repository under parquet-files topic.

• uber / petastorm

  Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

  deep-learningmachine-learningparquetparquet-filespyarrowpysparkpytorchsysmltensorflow
  Language:Python 1800

• Cinchoo / ChoETL

  ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

  csvparserwriterreaderflatxmljsonkeyvalueetlcinchoo-etletl-frameworkcsharpdotnetparquetparquet-filesyamlavro
  Language:C# 802
• parquet-go

  parquet-go / parquet-go

  High-performance Go package to read and write Parquet files

  apache-parquetcolumnar-formatgolangparquetparquet-filesperformance
  Language:Go 288

• mjakubowski84 / parquet4s

  Read and write Parquet in Scala. Use Scala classes as schema. No need to start a cluster.

  parquetscalahadoopakka-streamsakkastreamsreaderwriterparquet-filesawsgoogle-storagefs2bigdata
  Language:Scala 283

• uhussain / WebCrawlerForOnlineInflation

  Price Crawler - Tracking Price Inflation

  aws-athenasparkpython3s3-storagepandas-dataframeparquet-filescommoncrawlpetabytescalculate-inflation-ratesdash
  Language:Python 184

• hrbrmstr / sergeant

  :guardsman: Tools to Transform and Query Data with 'Apache' 'Drill'

  apache-drilldplyrdrillparquet-filesrr-cyberrstatssql
  Language:R 126

• PRQL / prql-query

  Query and transform data with PRQL

  csvcsv-convertercsv-filesdatadatabasedatafusionduckdbjsonjson-filesparquetparquet-filesparquet-toolsparquet-viewerpostgresqlprqlqueryquerydsl
  Language:Rust 125

• minio / spark-select

  A library for Spark DataFrame using MinIO Select API

  selectbigdatasparkspark-sqlamazon-s3parquet-filespysparksbtminio
  Language:Scala 96

• adrianulbona / osm-parquetizer

  A converter for the OSM PBFs to Parquet files

  apache-sparkconverteropenstreetmapparquet-filespbf
  Language:Java 92

• igor-suhorukov / openstreetmap_h3

  OSM planet dump high performance data loader. Transform OpenStreetMap World/Region PBF dump into partitioned by H3 regions PostGIS pgsnapshot (lossless) OSM schema representation and/or into ArrowIPC/Parquet dumps

  apach-sedonaapache-arrowapache-sparkarrowcitusdbcolumn-storeconverterduckdbgeometry-processinggeospatialjavaopenstreetmapparquetparquet-filespbfpbf-formatpostgispostgresqlworld
  Language:Java 92

• mongodb-labs / mongo-arrow

  MongoDB integrations for Apache Arrow. Export MongoDB documents to numpy array, parquet files, and pandas dataframes in one line of code.

  apache-arrowarrowmongodbnumpy-arrayspandas-dataframeparquet-filespython
  Language:Python 91

• hannes / miniparquet

  Library to read a subset of Parquet files

  parquetparquet-filesparquet-cppcppcpp11dependency-free
  Language:C++ 43

• strategicblue / parquet-floor

  A lightweight Java library that facilitates reading and writing Apache Parquet files without Hadoop dependencies

  parquet-filesjavaparquet
  Language:Java 41

• DataTech-Solutions / Threat-Detection-and-Visualization

  Threat Detection and Visualization

  apidatafactorydatalakedefenderdeltalakekeyvaultparquet-filespostmanpowerbirolebasedpermissionssccmserverlesssqlpoolservicenowsiemsqltenablescdedicatedsqlpoolactive-directoryazuremlstudiomachine-learning
  Language:TSQL 32

• grouzen / zio-apache-parquet

  Scala ZIO-powered Apache Parquet library

  apache-parquetbig-databigdataparquetparquet-filesparquet-formatparquet-toolsscalaziozio-streamszio2
  Language:Scala 21

• renesugar / FileConvert

  Converts between file formats such as CSV and Parquet

  csvapache-arrowapache-parquetparquet-filesconvertsparquet-cpp
  Language:C 14
• squey

  squey / squey

  Squey is a visualization software designed to interactively explore and understand large amounts of tabular data (this is the read-only mirror of https://gitlab.com/squey/squey)

  cybersecuritydata-analysisdata-visualizationparallel-coordinatespcapvisualizationexploratory-data-visualizationsparquetparquet-viewerparquet-filesdata-sciencetimeseriestimeseries-analysis
  Language:C++ 13

• vipinc007 / ParquetViewer

  A web application for viewing Apache Parquet files . This is a Python + Flask application

  parquet-filesparquet-viewerpyarrowpandasflask-applicationpython3
  Language:HTML 13

• trannguyenhan / tiki-data-analysis

  Streaming data of Tiki with Kafka and processing with Spark, visualize with Elasticsearch & Kibana.

  sparkspark-streamingkafkahadoopparquet-files
  Language:Java 11

• Foroozani / BigData_PySpark

  :bangbang: Handle Big Data for Machine Learning using Python and PySpark, Building ETL Pipelines with PySpark, MongoDB, and Bokeh

  big-dataapache-sparkpyspark-mllibpython3parquet-filesjsonpartitioned-datamongodb-database
  Language:Jupyter Notebook 8

• hrbrmstr / sergeant-caffeinated

  :guardsman: ☕️ Tools to Transform and Query Data with 'Apache' 'Drill'

  drillparquet-filesjdbcsqldplyrrrstats
  Language:R 7

• luusluus / vscode-parquet-visualizer

  vscode extension for SQL querying and visualizing parquet files

  explorerparquetparquet-filesparquet-viewerqueryviewervisualizervscode-extension
  Language:JavaScript 5

• rupeshtiwari / kafka-spark-streaming-avro-in-python

  Streaming kafka events using Spark in avro format and saving the events in parquet format

  kafkasparkpysparkavropythonpython3parquetparquet-filesconfluent-kafkascalajavaawsmskreal-time-streaming
  Language:Python 5

• alexkreidler / parquet2arrow

  A fast and simple command-line (CLI) tool to convert a Parquet file to an Apache Arrow file

  apache-arrowarrowclicommand-line-toolconverterparquetparquet-files
  Language:Rust 4

• Dorianteffo / vg-sales-glue-spark-terraform

  ETL job with AWS Glue

  awsaws-gluedockeriacparquet-filespythonsparkterraform
  Language:Python 4

• adrigrillo / NYCSparkTaxi

  Apache Spark application to get the top ten frequent routes and profitable areas

  big-datanycparquet-filespythonsparktaxi
  Language:Jupyter Notebook 3

• domvwt / parquet-inspector

  A command line tool for inspecting parquet files with PyArrow.

  cliparquetparquet-cliparquet-filesparquet-generatorparquet-toolsparquet-viewer
  Language:Python 3

• gpapag / spark-streaming-parquet

  Scala code to read Parquet files as streams in Spark Streaming using Avro.

  scalaparquetparquet-filessparkspark-streamingavro
  Language:Scala 3

• IgnacioMB / csvcli

  A light-weight command-line tool to browse and query CSV, Excel and Apache Parquet files, regardless of their size.

  csv-filesparquetdelimitercsvcliparquet-files
  Language:Python 3

• LouayMagdy / Weather-Stations-Monitoring

  DDIA Course Project

  bitcaskdesign-patternsdockerelasticsearchintegration-patternsjava-8k8skafkakibanaparquet-filespythonscalasparkstreaming
  Language:Java 3

• msigrupo / node-red-contrib-parquet

  Node-Red contrib that converts between a PARQUET string and its JavaScript object representation, in either direction.

  datasetnode-rednodejsparquetparquet-filesprocessing
  Language:HTML 3

• m-kwiedor / lambda-merge-parquet

  Merge Parquet Files on S3 with this AWS Lambda Function

  aws-s3fastparquetlambdaparquet-files
  Language:Python 2

• ostrokach / uniparc_xml_parser

  UniParc dataset describing ~300 million protein sequences converted into relational tables accessible through Google BigQuery (and as Parquet files).

  bioinformaticsprotein-sequencesprotein-domainsbigqueryparquet-filescsv-files
  Language:Rust 2

• OtavioHenrique / parquimetro

  Simple and small CLI to work with parquet files

  cliparquetparquet-filesparquet-toolsparquet-viewer
  Language:Go 2

• rigganni / AWS-Spark-Million-Song-ETL

  Load data from the Million Song Dataset into a final dimensional model stored in S3.

  aws-emraws-emr-clustersapache-sparkdimensional-modeletlparquetparquet-files
  Language:Python 2

• sudip-padhye / EDA-of-Malware-Infected-Devices-using-PySpark

  Explore factors associated with Malware Infection using Spark SQL

  pysparksparksqlparquet-filesrdd
  Language:Jupyter Notebook 2