Crtrpt / arrow-datafusion

Apache Arrow DataFusion SQL Query Engine

DataFusion

DataFusion is an extensible query planning, optimization, and execution framework, written in Rust, that uses Apache Arrow as its in-memory format.

Features

• SQL query planner with support for multiple SQL dialects
• DataFrame API
• Parquet, CSV, JSON, and Avro file formats are supported natively. Custom file formats can be supported by implementing a TableProvider trait.
• Supports popular object stores, including AWS S3, Azure Blob Storage, and Google Cloud Storage. There are extension points for implementing custom object stores.

Use Cases

DataFusion is modular in design with many extension points and can be used without modification as an embedded query engine and can also provide a foundation for building new systems. Here are some example use cases:

• DataFusion can be used as a SQL query planner and query optimizer, providing optimized logical plans that can then be mapped to other execution engines.
• DataFusion is used to create modern, fast and efficient data pipelines, ETL processes, and database systems, which need the performance of Rust and Apache Arrow and want to provide their users the convenience of an SQL interface or a DataFrame API.

Why DataFusion?

• High Performance: Leveraging Rust and Arrow's memory model, DataFusion achieves very high performance
• Easy to Connect: Being part of the Apache Arrow ecosystem (Arrow, Parquet and Flight), DataFusion works well with the rest of the big data ecosystem
• Easy to Embed: Allowing extension at almost any point in its design, DataFusion can be tailored for your specific use case
• High Quality: Extensively tested, both by itself and with the rest of the Arrow ecosystem, DataFusion can be used as the foundation for production systems.

DataFusion Community Extensions

There are a number of community projects that extend DataFusion or provide integrations with other systems.

Language Bindings

• datafusion-c
• datafusion-python
• datafusion-ruby
• datafusion-java

Integrations

• datafusion-bigtable
• datafusion-catalogprovider-glue
• datafusion-substrait

Known Uses

Here are some of the projects known to use DataFusion:

• Ballista Distributed SQL Query Engine
• Blaze Spark accelerator with DataFusion at its core
• CeresDB Distributed Time-Series Database
• Cloudfuse Buzz
• CnosDB Open Source Distributed Time Series Database
• Cube Store
• Dask SQL Distributed SQL query engine in Python
• datafusion-tui Text UI for DataFusion
• delta-rs Native Rust implementation of Delta Lake
• Flock
• InfluxDB IOx Time Series Database
• Parseable Log storage and observability platform
• qv Quickly view your data
• ROAPI
• Seafowl CDN-friendly analytical database
• Tensorbase
• VegaFusion Server-side acceleration for the Vega visualization grammar

(if you know of another project, please submit a PR to add a link!)

Example Usage

Please see the example usage in the user guide and the datafusion-examples crate for more information on how to use DataFusion.

Roadmap

Please see Roadmap for information of where the project is headed.

Architecture Overview

There is no formal document describing DataFusion's architecture yet, but the following presentations offer a good overview of its different components and how they interact together.

• (July 2022): DataFusion and Arrow: Supercharge Your Data Analytical Tool with a Rusty Query Engine: recording and slides
• (March 2021): The DataFusion architecture is described in Query Engine Design and the Rust-Based DataFusion in Apache Arrow: recording (DataFusion content starts ~ 15 minutes in) and slides
• (February 2021): How DataFusion is used within the Ballista Project is described in *Ballista: Distributed Compute with Rust and Apache Arrow: recording

User Guide

Please see User Guide for more information about DataFusion.

Contributor Guide

Please see Contributor Guide for information about contributing to DataFusion.