fwessels / json2arrow

Convert JSON files to Apache Arrow.

JSON to Arrow

Convert JSON files to Apache Arrow. You may also be interested in csv2arrow, json2parquet, or csv2parquet.

Installation

Download prebuilt binaries

You can get the latest releases from https://github.com/domoritz/json2arrow/releases/.

With Cargo

cargo install json2arrow

Usage

USAGE:
    json2arrow [FLAGS] [OPTIONS] <JSON> [ARROW]

ARGS:
    <JSON>     Input JSON file
    <ARROW>    Output file, stdout if not present

FLAGS:
    -h, --help       Prints help information
    -v, --verbose    Print the schema to stderr
    -V, --version    Prints version information

OPTIONS:
    -m, --max-read-records <max-read-records>
            The number of records to infer the schema from. All rows if not present

Limitations

Since we use teh Arrow JSON loader, we are limited to what it supports. Right now, it supports JSON line-delimited files.

{ "a": 42, "b": true }
{ "a": 12, "b": false }
{ "a": 7, "b": true }

For Developers

To format the code, run

cargo clippy && cargo fmt