andrusha / ratatool

A tool for data sampling, data generation, and data diffing

Ratatool

A tool for random data sampling and generation

Features

• ScalaCheck Generators - ScalaCheck generators (Gen[T]) for property-based testing for Avro, Protocol Buffers and BigQuery TableRow
• IO - utilities for reading and writing records in Avro, Parquet (via Avro GenericRecord), BigQuery and TableRow JSON files. Local file system, HDFS and Google Cloud Storage are supported.
• Samplers - random data samplers for Avro, BigQuery and Parquet. True random sampling is supported for Avro only while head mode (sampling from the start) is supported for all sources.
• Diffy - field-level record diff tool for Avro, Protobuf and BigQuery TableRow.
• BigDiffy - Scio library for pairwise field-level statistical diff of data sets. See slides for more.
• Command line tool - command line tool for local sampler, or executing BigDiffy and BigSampler.
• Shapeless - An extension for Case Class Diffing via Shapeless.

For more information or documentation, project level READMEs are provided.

Usage

If you use sbt add the following dependency to your build file:

libraryDependencies += "com.spotify" %% "ratatool-scalacheck" % "0.3.10" % "test"

If needed, the following other libraries are published:

• ratatool-diffy
• ratatool-sampling

Or install via our Homebrew tap if you're on a Mac:

brew tap spotify/public
brew install ratatool
ratatool

Or download the release jar and run it.

wget https://github.com/spotify/ratatool/releases/download/v0.3.10/ratatool-cli-0.3.10.tar.gz
bin/ratatool directSampler

The command line tool can be used to sample from local file system or Google Cloud Storage directly if Google Cloud SDK is installed and authenticated.

bin/ratatool bigSampler avro --head -n 1000 --in gs://path/to/dataset --out out.avro
bin/ratatool bigSampler parquet --head -n 1000 --in gs://path/to/dataset --out out.parquet

# write output to both JSON file and BigQuery table
bin/ratatool bigSampler bigquery --head -n 1000 --in project_id:dataset_id.table_id \
    --out out.json--tableOut project_id:dataset_id.table_id

It can also be used to sample from HDFS with if core-site.xml and hdfs-site.xml are available.

bin/ratatool bigSampler avro \
    --head -n 10 --in hdfs://namenode/path/to/dataset --out file:///path/to/out.avro

Or execute BigDiffy directly

bin/ratatool bigDiffy --mode avro --key record.key \
    --lhs gs://path/to/left --rhs gs://path/to/right --output gs://path/to/output \
    --runner DataflowRunner ....

License

Copyright 2016-2018 Spotify AB.

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0