parquet

There are 13 repositories under parquet topic.

• multiprocessio / dsq

  Commandline tool for running SQL queries against JSON, CSV, Excel, Parquet, and more.

  clicsvexcelgolangjsonopenoffice-calcparquetsqltsv
  Language:Go 3693

• roapi / roapi

  Create full-fledged APIs for slowly moving datasets without writing a single line of code.

  sqlgraphqlarrowrest-apianalyticsquerycolumnarrustin-memory-databasedatafusionblob-storagecloud-nativeparquetquery-frontendsdatasetsstatic-datasetss3delta-lake
  Language:Rust 3160

• apache / parquet-java

  Apache Parquet Java

  apacheparquetparquet-java
  Language:Java 2505
• qsv

  jqnatividad / qsv

  CSVs sliced, diced & analyzed.

  csvdata-wranglingtsvcliopendatadata-engineeringckanexcelluaudatapackageparquetpostgresqlsqlitepolarssnappyrustsqlgeocodetimeseries
  Language:Rust 2336

• apache / drill

  Apache Drill is a distributed MPP query layer for self describing data

  big-datadrillhadoophivejavajdbcparquetsql
  Language:Java 1925

• uber / petastorm

  Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

  deep-learningmachine-learningparquetparquet-filespyarrowpysparkpytorchsysmltensorflow
  Language:Python 1766

• gchq / Gaffer

  A large-scale entity and relation database supporting aggregation of properties

  accumulographgraph-databasehadoopbig-dataaggregationhbaseparquetspark
  Language:Java 1751

• apache / parquet-format

  Apache Parquet Format

  apacheparquetparquet-format
  Language:Thrift 1701
• rill

  rilldata / rill

  Rill is a tool for effortlessly transforming data sets into powerful, opinionated dashboards using SQL. BI-as-code.

  duckdbsveltesveltejssveltekitdatavizcsvparquetparquet-toolsgolangs3gcsdata-analysissqlbidatadata-visualizationparquet-viewerbusiness-analyticssql-editor
  Language:Go 1562

• quiltdata / quilt

  Quilt is a data mesh for connecting people with actionable data

  datadata-engineeringdata-version-controldata-versioningpythonserializationparquet
  Language:Jupyter Notebook 1317

• paradigmxyz / cryo

  cryo is the easiest way to extract blockchain data to parquet, csv, json, or python dataframes

  cryptoethereumevmparquetrust
  Language:Rust 1049

• bigdatagenomics / adam

  ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark, and Apache Parquet. Apache 2 licensed.

  avrobig-databioinformaticsgenomicsjavaparquetpythonrscalaspark
  Language:Scala 968

• Cinchoo / ChoETL

  ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

  csvparserwriterreaderflatxmljsonkeyvalueetlcinchoo-etletl-frameworkcsharpdotnetparquetparquet-filesyamlavro
  Language:C# 759

• HariSekhon / DevOps-Python-tools

  80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

  cloudformationpythonhbasejsonavroparquetsparkpysparktravis-cielasticsearchsolrhadoophdfsdockerhubdockerlinuxawsdevopsgcpgcf
  Language:Python 745

• mukunku / ParquetViewer

  Simple Windows desktop application for viewing & querying Apache Parquet files

  apache-parquetbig-datadot-netparquetwindows-desktop
  Language:C# 706
• kglab

  DerwenAI / kglab

  Graph Data Science: an abstraction layer in Python for building knowledge graphs, integrated with popular graph libraries – atop Pandas, NetworkX, RAPIDS, RDFlib, pySHACL, PyVis, morph-kgc, pslpython, pyarrow, etc.

  knowledge-graphrdflibnetworkxskosparquetgraph-algorithmspyvisjson-ldsparqlshaclprobabilistic-soft-logicpandasowlinferencepython-igraphgraph-librariesrapidsroamresearchgraph-thinkingr2rml-mapping
  Language:Jupyter Notebook 567
• pystore

  ranaroussi / pystore

  Fast data store for Pandas time-series data

  daskdatabasedataframedatastorepandasparquettimeseries
  Language:Python 545
• vscode-data-preview

  RandomFractals / vscode-data-preview

  Data Preview 🈸 extension for importing 📤 viewing 🔎 slicing 🔪 dicing 🎲 charting 📊 & exporting 📥 large JSON array/config, YAML, Apache Arrow, Avro, Parquet & Excel data files

  vscodeextensioncsvarrowdataperspectiveviewerjsonarrayconfigyamlexcelavroparquet
  Language:TypeScript 540
• lonboard

  developmentseed / lonboard

  A Python library for fast, interactive geospatial vector data visualization in Jupyter.

  anywidgetapache-arrowapache-parquetdata-visualizationdeck-glgeoarrowgeopandasgeoparquetgeospatialgeospatial-analysisjupyterjupyter-widgetlongboardmap-visualizationmapsparquetpythonvisualizationwebgl
  Language:Python 510

• kylebarron / parquet-wasm

  Rust-based WebAssembly bindings to read and write Apache Parquet data

  webassemblywasmrustparquetjavascriptarrowapache-arrowapache-parquet
  Language:Rust 483

• Netflix / iceberg

  Iceberg is a table format for large, slow-moving tabular data

  sparkhadoopparquetavro
  Language:Java 469

• apache / parquet-cpp

  Apache Parquet

  parquetjavabig-data
  439

• skale-me / skale

  High performance distributed data processing engine

  nodejsclusteraws-s3azure-storageparquetmachine-learningskale
  Language:JavaScript 399

• moshe / elasticsearch_loader

  A tool for batch loading data files (json, parquet, csv, tsv) into ElasticSearch

  csvelasticsearchelasticsearch-loaderjsonlogstashparquetpython
  Language:Python 397

• jorgecarleitao / parquet2

  Fastest and safest Rust implementation of parquet. `unsafe` free. Integration-tested against pyarrow

  parquetsafeparallelismrust
  Language:Rust 349

• ironSource / parquetjs

  fully asynchronous, pure JavaScript implementation of the Parquet file format

  nodejsjavascriptparquet
  Language:JavaScript 346

• segmentio / parquet-go

  Go library to read/write Parquet files

  gogolangparquetperformance
  Language:Go 341

• spotify / ratatool

  A tool for data sampling, data generation, and data diffing

  scalascalacheckavroparquetbigqueryprotobuf
  Language:Scala 338

• sksamuel / centurion

  Kotlin Bigdata Toolkit

  kotlinjavabigdataparquetorc
  Language:Kotlin 326

• Eugene-Mark / bigdata-file-viewer

  A cross-platform (Windows, MAC, Linux) desktop application to view common bigdata binary format like Parquet, ORC, AVRO, etc. Support local file system, HDFS, AWS S3, Azure Blob Storage ,etc.

  bigdataparquetorcavrohdfs
  Language:Java 282

• mjakubowski84 / parquet4s

  Read and write Parquet in Scala. Use Scala classes as schema. No need to start a cluster.

  parquetscalahadoopakka-streamsakkastreamsreaderwriterparquet-filesawsgoogle-storagefs2bigdata
  Language:Scala 282

• fraugster / parquet-go

  Go package to read and write parquet files. parquet is a file format to store nested data structures in a flat columnar data format. It can be used in the Hadoop ecosystem and with tools such as Presto and AWS Athena.

  golanggolang-packageparquetparquet-schemaathenaprestohadoophacktoberfest
  Language:Go 281

• grai-io / grai-core

  datadata-lineagedata-sciencedataengineeringdatalineagedbtdjangofivetranhacktoberfestmssqlmysqlopen-sourceparquetpostgresqlpythonredshiftsnowflake
  Language:Python 281
• pqrs

  manojkarthick / pqrs

  Command line tool for inspecting Parquet files

  parquetarrowrust
  Language:Rust 268

• cldellow / sqlite-parquet-vtable

  A SQLite vtable extension to read Parquet files

  parquetsqlitesqlite3apache-arrowapache-parquet
  Language:C++ 266
• amazon-s3-find-and-forget

  awslabs / amazon-s3-find-and-forget

  Amazon S3 Find and Forget is a solution to handle data erasure requests from data lakes stored on Amazon S3, for example, pursuant to the European General Data Protection Regulation (GDPR)

  data-lakeamazon-s3s3gdprawsparquetdata-erasureright-to-be-forgottenccpabig-dataprivacydata
  Language:Python 236