parquet

There are 12 repositories under parquet topic.

multiprocessio / dsq
Commandline tool for running SQL queries against JSON, CSV, Excel, Parquet, and more.
cli csv excel golang json openoffice-calc parquet sql tsv
Language:Go 3617
roapi / roapi
Create full-fledged APIs for slowly moving datasets without writing a single line of code.
sql graphql arrow rest-api analytics query columnar rust in-memory-database datafusion blob-storage cloud-native parquet query-frontends datasets static-datasets s3 delta-lake
Language:Rust 3070
apache / parquet-mr
Apache Parquet
big-data java parquet
Language:Java 2401
qsv
jqnatividad / qsv
CSVs sliced, diced & analyzed.
ckan cli csv data-engineering data-wrangling datapackage excel geocode luau opendata parquet polars postgresql rust snappy sql sqlite timeseries tsv
Language:Rust 2214
apache / drill
Apache Drill is a distributed MPP query layer for self describing data
java big-data drill sql hive hadoop jdbc parquet
Language:Java 1891
uber / petastorm
Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
deep-learning machine-learning parquet parquet-files pyarrow pyspark pytorch sysml tensorflow
Language:Python 1748
gchq / Gaffer
A large-scale entity and relation database supporting aggregation of properties
accumulo graph graph-database hadoop big-data aggregation hbase parquet spark
Language:Java 1731
apache / parquet-format
Apache Parquet
big-data java parquet
Language:Java 1636
rill
rilldata / rill
Rill is a tool for effortlessly transforming data sets into powerful, opinionated dashboards using SQL. BI-as-code.
bi business-analytics csv data data-analysis data-visualization dataviz duckdb gcs golang parquet parquet-tools parquet-viewer s3 sql sql-editor svelte sveltejs sveltekit
Language:Go 1337
quiltdata / quilt
Quilt is a data mesh for connecting people with actionable data
data data-engineering data-version-control data-versioning parquet python serialization
Language:Jupyter Notebook 1311
paradigmxyz / cryo
cryo is the easiest way to extract blockchain data to parquet, csv, json, or python dataframes
crypto ethereum evm parquet rust
Language:Rust 971
bigdatagenomics / adam
ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark, and Apache Parquet. Apache 2 licensed.
avro big-data bioinformatics genomics java parquet python r scala spark
Language:Scala 966
Cinchoo / ChoETL
ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)
csv parser writer reader flat xml json keyvalue etl cinchoo-etl etl-framework csharp dotnet parquet parquet-files yaml avro
Language:C# 735
HariSekhon / DevOps-Python-tools
80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.
cloudformation python hbase json avro parquet spark pyspark travis-ci elasticsearch solr hadoop hdfs dockerhub docker linux aws devops gcp gcf
Language:Python 729
mukunku / ParquetViewer
Simple windows desktop application for viewing & querying Apache Parquet files
apache-parquet big-data dot-net parquet windows-desktop
Language:C# 637
kglab
DerwenAI / kglab
Graph Data Science: an abstraction layer in Python for building knowledge graphs, integrated with popular graph libraries – atop Pandas, NetworkX, RAPIDS, RDFlib, pySHACL, PyVis, morph-kgc, pslpython, pyarrow, etc.
knowledge-graph rdflib networkx skos parquet graph-algorithms pyvis json-ld sparql shacl probabilistic-soft-logic pandas owl inference python-igraph graph-libraries rapids roamresearch graph-thinking r2rml-mapping
Language:Jupyter Notebook 554
pystore
ranaroussi / pystore
Fast data store for Pandas time-series data
datastore dask parquet pandas timeseries database dataframe
Language:Python 538
vscode-data-preview
RandomFractals / vscode-data-preview
Data Preview 🈸 extension for importing 📤 viewing 🔎 slicing 🔪 dicing 🎲 charting 📊 & exporting 📥 large JSON array/config, YAML, Apache Arrow, Avro, Parquet & Excel data files
vscode extension csv arrow data perspective viewer json array config yaml excel avro parquet
Language:TypeScript 522
Netflix / iceberg
Iceberg is a table format for large, slow-moving tabular data
spark hadoop parquet avro
Language:Java 465
apache / parquet-cpp
Apache Parquet
big-data java parquet
Language:C++ 436
skale-me / skale
High performance distributed data processing engine
nodejs cluster aws-s3 azure-storage parquet machine-learning skale
Language:JavaScript 397
moshe / elasticsearch_loader
A tool for batch loading data files (json, parquet, csv, tsv) into ElasticSearch
python elasticsearch parquet csv json elasticsearch-loader logstash
Language:Python 395
lonboard
developmentseed / lonboard
Python library for fast, interactive geospatial vector data visualization in Jupyter.
anywidget apache-arrow deck-gl geoarrow geopandas jupyter jupyter-widget longboard map-visualization data-visualization geospatial-analysis maps python visualization webgl geoparquet apache-parquet parquet geospatial
Language:Python 379
jorgecarleitao / parquet2
Fastest and safest Rust implementation of parquet. `unsafe` free. Integration-tested against pyarrow
parallelism parquet rust safe
Language:Rust 347
ironSource / parquetjs
fully asynchronous, pure JavaScript implementation of the Parquet file format
javascript nodejs parquet
Language:JavaScript 342
spotify / ratatool
A tool for data sampling, data generation, and data diffing
avro bigquery parquet protobuf scala scalacheck
Language:Scala 337
segmentio / parquet-go
Go library to read/write Parquet files
go golang parquet performance
Language:Go 336
sksamuel / kotlin-big-data
Kotlin Bigdata Toolkit
bigdata java kotlin orc parquet
Language:Kotlin 326
Eugene-Mark / bigdata-file-viewer
A cross-platform (Windows, MAC, Linux) desktop application to view common bigdata binary format like Parquet, ORC, AVRO, etc. Support local file system, HDFS, AWS S3, Azure Blob Storage ,etc.
avro bigdata hdfs orc parquet
Language:Java 281
fraugster / parquet-go
Go package to read and write parquet files. parquet is a file format to store nested data structures in a flat columnar data format. It can be used in the Hadoop ecosystem and with tools such as Presto and AWS Athena.
golang golang-package parquet parquet-schema athena presto hadoop hacktoberfest
Language:Go 279
mjakubowski84 / parquet4s
Read and write Parquet in Scala. Use Scala classes as schema. No need to start a cluster.
parquet scala hadoop akka-streams akka streams reader writer parquet-files aws google-storage fs2 bigdata
Language:Scala 270
grai-io / grai-core
hacktoberfest data datalineage dbt open-source postgresql python mssql mysql snowflake data-science dataengineering fivetran django parquet redshift data-lineage
Language:Python 267
cldellow / sqlite-parquet-vtable
A SQLite vtable extension to read Parquet files
apache-arrow apache-parquet parquet sqlite sqlite3
Language:C++ 263
pqrs
manojkarthick / pqrs
Command line tool for inspecting Parquet files
parquet arrow rust
Language:Rust 245
amazon-s3-find-and-forget
awslabs / amazon-s3-find-and-forget
Amazon S3 Find and Forget is a solution to handle data erasure requests from data lakes stored on Amazon S3, for example, pursuant to the European General Data Protection Regulation (GDPR)
data-lake amazon-s3 s3 gdpr aws parquet data-erasure right-to-be-forgotten ccpa big-data privacy data
Language:Python 232
scikit-hep / awkward-0.x
Manipulate arrays of complex data structures as easily as Numpy.
python python3 numpy big-data analysis columnar columnar-storage apache-arrow arrow parquet hdf5 root root-cern scikit-hep
Language:Python 216

parquet

multiprocessio / dsq

roapi / roapi

apache / parquet-mr

jqnatividad / qsv

apache / drill

uber / petastorm

gchq / Gaffer

apache / parquet-format

rilldata / rill

quiltdata / quilt

paradigmxyz / cryo

bigdatagenomics / adam

Cinchoo / ChoETL

HariSekhon / DevOps-Python-tools

mukunku / ParquetViewer

DerwenAI / kglab

ranaroussi / pystore

RandomFractals / vscode-data-preview

Netflix / iceberg

apache / parquet-cpp

skale-me / skale

moshe / elasticsearch_loader

developmentseed / lonboard

jorgecarleitao / parquet2

ironSource / parquetjs

spotify / ratatool

segmentio / parquet-go

sksamuel / kotlin-big-data

Eugene-Mark / bigdata-file-viewer

fraugster / parquet-go

mjakubowski84 / parquet4s

grai-io / grai-core

cldellow / sqlite-parquet-vtable

manojkarthick / pqrs

awslabs / amazon-s3-find-and-forget

scikit-hep / awkward-0.x