pest. The Elegant Parser
pest is a PEG parser with simplicity and speed in mind.
Elegant grammar
Defining a grammar for a list of alpha-numeric identifiers where the first identifier does not start with a digit is as straight-forward as:
alpha = { 'a'..'z' | 'A'..'Z' }
digit = { '0'..'9' }
ident = { (alpha | digit)+ }
ident_list = _{ !digit ~ ident ~ (" " ~ ident)+ }
// ^
// ident_list rule is silent which means it produces no tokens
This is then saved in a .pest
grammar file and is never mixed up with Rust code which results in an always up-to-date
formal definition of the grammar which is very easy to maintain.
Pairs API
The grammar can be used to derive a Parser
implementation automatically. Parsing returns an iterator of nested token
pairs:
extern crate pest;
#[macro_use]
extern crate pest_derive;
use pest::Parser;
#[derive(Parser)]
#[grammar = "ident.pest"]
struct IdentParser;
fn main() {
let pairs = IdentParser::parse(Rule::ident_list, "a1 b2").unwrap_or_else(|e| panic!("{}", e));
// Because ident_list is silent, the iterator will contain idents
for pair in pairs {
let span = pair.clone().into_span();
// A pair is a combination of the rule which matched and a span of input
println!("Rule: {:?}", pair.as_rule());
println!("Span: {:?}", span);
println!("Text: {}", span.as_str());
// A pair can be converted to an iterator of the tokens which make it up:
for inner_pair in pair.into_inner() {
let inner_span = inner_pair.clone().into_span();
match inner_pair.as_rule() {
Rule::alpha => println!("Letter: {}", inner_span.as_str()),
Rule::digit => println!("Digit: {}", inner_span.as_str()),
_ => unreachable!()
};
}
}
}
This produces the following output:
Rule: ident
Span: Span { start: 0, end: 2 }
Text: a1
Letter: a
Digit: 1
Rule: ident
Span: Span { start: 3, end: 5 }
Text: b2
Letter: b
Digit: 2
Meaningful error reporting
Parsing "123"
instead of "a1 b2"
in the code above will result in the following panic:
thread 'main' panicked at ' --> 1:1
|
1 | 123
| ^---
|
= unexpected digit', src/main.rs:12
while parsing "ab *"
will result in:
thread 'main' panicked at ' --> 1:4
|
1 | ab *
| ^---
|
= expected ident', src/main.rs:12
Sheer performance
pest provides parsing performance in the same league as carefully written manual parsers. The following JSON benchmark puts it somewhere in between one of the most optimized JSON parsers, ujson4c, and a static native-speed parser, nom.
The first entry of pest scores 36ms, while the second scores 96ms since it's mapping Pair
s
to a custom JSON AST. While the first entry forms a perfectly usable tree, it does not process
the file to a fully-processed JSON object. The second one does, but since it has an extra
intermediate representation of the object, it repeats some work.
The benchmark uses a large 2MB JSON file. Tested on a 2.6GHz Intel® Core™ i5 running macOS.
Other features
- precedence climbing
- input handling
- custom errors
- runs on stable Rust
Usage
pest requires Cargo and Rust 1.23.
Add the following to Cargo.toml
:
pest = "^1.0"
pest_derive = "^1.0"
and in your Rust lib.rs
or main.rs
:
extern crate pest;
#[macro_use]
extern crate pest_derive;
Projects using pest
- pest_meta (bootstrapped)
- brain
- Chelone
- comrak
- graphql-parser
- handlebars-rust
- Huia
- py_literal
- rouler
- RuSh
- rs_pbrt
- stache
- tera
- ui_gen
- ukhasnet-parser
Special thanks
A special round of applause goes to prof. Marius Minea for his guidance and all pest contributors, some of which being none other than my friends.