pyarrow

There are 1 repository under pyarrow topic.

• vaexio / vaex

  Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀

  dataframepythonbigdatatabular-datavisualizationmemory-mapped-filehdf5machine-learningmachinelearningdata-sciencepyarrow
  Language:Python 8444

• ibis-project / ibis

  the portable Python dataframe library

  bigqueryclickhousedatabasedatafusionduckdbimpalamssqlmysqlpandaspolarspostgresqlpyarrowpysparkpythonsnowflakesqlsqlitetrino
  Language:Python 6201

• uber / petastorm

  Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

  tensorflowpytorchdeep-learningmachine-learningsysmlpysparkpyarrowparquetparquet-files
  Language:Python 1867

• narwhals-dev / narwhals

  Lightweight and extensible compatibility layer between dataframe libraries!

  cudfibispandaspolarspyarrowdaskduckdbpyspark
  Language:Python 1358

• wheretrue / biobear

  Work with bioinformatic files using Arrow, Polars, and/or DuckDB

  bioinformaticsbiologyrust-biosamtoolsarrowpythonbiopythonduckdbpolarspyarrow
  Language:Rust 190

• dacort / faker-cli

  Command-line interface to quickly generate fake CSV and JSON data

  awscsvfaker-providerjsondeltalakeparquetpyarrow
  Language:Python 72

• zen-xu / pyarrow-stubs

  Type annotations for pyarrow

  pyarrowtyping
  Language:Python 42
• chicago-crimes

  RandomFractals / chicago-crimes

  Exploring Chicago crimes dataset with Jupyter notebooks, DuckDB, Malloy and new Panel/PyScript data and dashboard tools.

  chicagocrimesjupyter-notebookspolarspyarrowparquetjulialarge-csvduckdbmalloymalloydata
  Language:Jupyter Notebook 38

• kraina-ai / overturemaestro

  An open-source tool for reading OvertureMaps data with multiprocessing and additional Quality-of-Life features

  geogeospatialopen-sourceopenstreetmapoverture-mapsoverturemapspyarrowpython
  Language:Python 31

• icaropires / pdf2dataset

  Converts a whole subdirectory with a big (or small) volume of PDF documents to a dataset (pandas DataFrame) with error tracking and choice of features

  python3raydistributed-systemsdistributed-computingparallelpdfdata-scienceparquettesseract-ocrtesseractocrpytesseractpytesseract-ocrpdf2imagepdftotextpythonpandas-dataframepyarrow
  Language:Python 20

• thread53 / pqviewer

  View Apache Parquet Files In Your Terminal

  parquetpyarrowpythontextualterminalcli
  Language:Python 18

• ismailhammounou / db2ixf

  db2ixf is a python package with a CLI that simplifies the parsing and processing of IBM Integration eXchange Format (IXF) files.

  conversionconvertercsvdb2db2-databaseibmibm-cloudixfjsonparquetparserparsingparsing-libraryprocessingpythonclideltalakepolarspyarrowjsonlines
  Language:Python 16

• milesgranger / flaco

  (PoC) A very memory-efficient way to read data from PostgreSQL

  postgresqlrustpythonarrowpyarrow
  Language:Rust 15

• vipinc007 / ParquetViewer

  A web application for viewing Apache Parquet files . This is a Python + Flask application

  parquet-filesparquet-viewerpyarrowpandasflask-applicationpython3
  Language:HTML 13

• SaelKimberly / rxls

  Reading both XLSX and XLSB files, fast and memory-safe, with Python, into PyArrow

  pyarrow
  Language:Jupyter Notebook 11

• DanielAvdar / pandas-pyarrow

  Seamlessly switch Pandas DataFrame backend to PyArrow.

  arrowbackenddtypespandaspandas-dataframepyarrowpythondb-dtypespandas-pyarrowpandas-arrow
  Language:Python 9

• legout / pydala

  Poor mans simple python api for creating a local or remote datalake based on several (pyarrow) datasets using duckdb

  pyarrowdatalakeduckdb
  Language:Python 9

• trustedshops-public / schema2pyarrow

  Converts AsyncApi and JsonSchema to PyArrow schema

  asyncapidata-engineeringdatacontractsjsonschemapyarrowschematslibraries
  Language:Python 9

• goalzz85 / sql2arrow

  SQL2Arrow, short for 'SQL to Arrow,' is a Python library that provides convenient and high-performance methods to parse INSERT SQL statements into Arrow arrays. It is particularly useful for analyzing data dumped by mysqldump or other tools.

  arrowicebergmysqlpyarrowrustsqlpippostgresqlpython
  Language:Rust 7

• legout / pydala2

  poor man´s data lake - Simple api to efficiently query your parquet datasets using Duckdb or polars

  duckdbfsspeclocallocalcacheobject-storagepandaspolarspyarrowpython
  Language:Python 6

• lykmapipo / Python-Spark-Log-Analysis

  Python scripts to process, and analyze log files using PySpark.

  apache-arrowdata-analysisdata-extractiondata-processingdata-transformationlog-analysislog-analyzerlog-monitorlykmapipopandaspyarrowpysparkpythonseabornapache-sparkapache-spark-sqlsqlspark-nlpsparkml-pipelinesspark-ml
  Language:Python 6

• asierra01 / pyarrow_to_db2

  ibm_db extension to load a pyarrow table to db2

  pyarrowdb2luwpython3
  Language:C 5

• lykmapipo / NYC-TLC-Trip-Data

  Python scripts to download, process, and analyze the New York City Taxi and Limousine Commission (TLC) Trip Record Data dataset

  datadata-engineeringdata-extractiondata-transformationetllykmapipometadatanycpythongeopandaspandaspyarrows3apache-arrowjupyterlabjoblibfsspecapache-sparknyc-taxi-dataset
  Language:Jupyter Notebook 5

• dominiquegarmier / oakstore

  highspeed timeseries pandas dataframe database

  big-datadaskdatabasepandasparquetpyarrowtimeseriesdeep-learningfinancemachine-learningdata-sciencedatasetdatawarehousedeeplearningpython
  Language:Python 4

• dr-saad-la / Pyarrow-Tuts

  Pyarrow Tutorials

  programmingpyarrowpython3tutorials
  Language:Jupyter Notebook 4

• jaysnm / dremio-arrow

  Dremio Arrow Flight Client

  dataframedremiodremio-arrowpandaspyarrowpythonr
  Language:Python 4

• xbrianh / xdlake

  A loose implementation of the deltalake protocol, written in Python on top of pyarrow, focused on extensibility, customizability, and distributed data.

  databricksdelta-lakedeltalakedeltatableshiveparquetpyarrowpythonspark
  Language:Python 4

• kiwi0fruit / featherhelper

  Concise interface to cache numpy arrays and pandas dataframes

  pythonnumpypandascachepyarrow
  Language:Python 3

• psmyth94 / biosets

  A bioinformatics extension of 🤗 Datasets library, built for ML applications on biological and omics data, offering easy integration of metadata and low-code data management tools.

  big-databioinfoclassificationdata-preprocessingdata-processingdata-sciencedatasetsgenomicshigh-performancehuggingfacemachine-learningmetadataomicsopen-sourcepandaspolarsproteomicspyarrowpythonregression
  Language:Python 3

• d-chris / federleicht

  lightweigth function decorators to cache your `pandas.DataFrame` as feather.

  cachepandaspyarrowpypi-packagexxhash
  Language:Python 2

• namansnghl / SQLify

  Text (biz req) to SQL Semantic Parser with LLMs Transfer Learning. This will help Analysts query DB without knowing SQL.

  bartnmt-modelpyarrowt5-smalldatabases
  Language:Jupyter Notebook 2

• No-Country-simulation / c22-29-ft-data-bi

  🚀Optimización del control de inventario para BottleFlow Logistics: un enfoque estratégico basado en datos #Supply Chain🚀

  google-cloud-platformmysqlpowerbipyarrowpythonquartotrellogoogle-cloud-sql
  Language:Jupyter Notebook 2

• 345950647 / clickhouse_types

  Converting ClickHouse types into other schemas' types

  clickhousepyarrowsqlalchemy
  Language:Python 1

• anto18671 / arrow-datasets

  A high-performance Rust utility that converts large image datasets into chunked Apache Arrow files for efficient storage and processing.

  arrowdatasetshuggingfaceimage-datasetpreprocessingpyarrow
  Language:Rust 1

• BenyaminZojaji / mongodb_tutorial

  MongoDB tutorial repository

  mongodbpyarrowpythontutorial
  Language:Python 1

• edisedis777 / Coffee-Shops-Analysis

  This project analyzes the Foursquare Open Source Places dataset to explore the distribution of coffee shops across the United States, with a special focus on Portland, Oregon.

  coffeecoffee-shopdaftpythonaltairpolarsfoliumplotlypyarrow
  Language:HTML 1