JSON-e
JSON-e is a data-structure parameterization system for embedding context in JSON objects.
The central idea is to treat a data structure as a "template" and transform it, using another data structure as context, to produce an output data structure.
There are countless libraries to do this with strings, such as mustache. What makes JSON-e unique is that it operates on data structures, not on their textual representation. This allows input to be written in a number of formats (JSON, YAML, etc.) or even generated dynamically. It also means that the output cannot be "invalid", even when including large chunks of contextual data.
JSON-e is also designed to be safe for use on untrusted data. It never uses
eval
or any other function that might result in arbitrary code execution. It
also disallows unbounded iteration, so any JSON-e rendering operation will
finish in finite time.
Changes
See CHANGELOG.rst for the changes in each version of this library.
Interface
JavaScript
The JS module is installed with either of
npm install --save json-e
yarn add json-e
The module exposes following interface:
import jsone from 'json-e';
var template = {a: {$eval: "foo.bar"}};
var context = {foo: {bar: "zoo"}};
console.log(jsone(template, context));
// -> { a: 'zoo' }
Note that the context can contain functions, and those functions can be called from the template:
var template = {$eval: "foo(1)"};
var context = {"foo": function(x) { return x + 2; }};
console.log(jsone(template, context)); // -> 3
NOTE: Context functions are called synchronously. Any complex asynchronous operations should be handled before rendering the template.
NOTE: If the template is untrusted, it can pass arbitrary data to functions in the context, which must guard against such behavior.
Browser
JSON-e is distributed as a CommonJS package is not designed to be included
directly in a browser with <script>
. Instead, it must be incorporated using a
tool that understands CommonJS such as Webpack. See
Neutrino for an easy, configuration-free way to
build such applications.
TypeScript
The JS module is installed with either of
npm install --save json-e
yarn add json-e
Note: Type definitions are included with this package, so there's no need of seperate
@types/..
installation.
As 'json-e' is a CommonJS module, the package must be imported like this (more..) for type definitions to work properly:
import jsone = require('json-e');
var template = {a: {$eval: "foo.bar"}};
var context = {foo: {bar: "zoo"}};
console.log(jsone(template, context));
// -> { a: 'zoo' }
Python
The Python distribution is installed with
pip install json-e
The distribution exposes a render
function:
import jsone
template = {"a": {"$eval": "foo.bar"}}
context = {"foo": {"bar": "zoo"}}
print(jsone.render(template, context)) # -> {"a": "zoo"}
and also allows custom functions in the context:
template = {"$eval": "foo(1)"}
context = {"foo": lambda x: x + 2}
print(jsone.render(template, context)) # -> 3
Go (golang)
The golang package for json-e exposes a Render
function:
import (
"fmt"
"github.com/taskcluster/json-e"
)
// Template must be given using types:
// map[string]interface{}, []interface{}, float64, string, bool, nil
// The same types that json.Unmarshal() will create when targeting an interface{}
template := map[string]interface{}{
"result": map[string]interface{}{
"$eval": "f() + 5",
},
}
// Context can be JSON types just like template, but may also contain functions
// these can JSON types as arguments, and return a value and optionally an error.
context := map[string]interface{}{
"f": func() int { return 37 },
}
func main() {
value, err := jsone.Render(template, context)
fmt.Printf("%#v\n", value)
}
Third-Party Integrations
rjsone
You can use the 3rd party package rjsone to template JSON-e from the command line, passing templates/contexts as files or arguments and using stdout for the result.
Bazel
You can use 3rd party Bazel rule to invoke rjsone (see above) from Bazel build files.
Terraform
The jsone Terraform provider allows use of JSON-e for templating objects within Terraform.
Language Reference
The examples here are given in YAML for ease of reading. Of course, the rendering operation takes place on the parsed data, so the input format is irrelevant to its operation.
Simple Operations
All JSON-e directives involve the $
character, so a template without any directives is
rendered unchanged:
template: {key: [1,2,{key2: 'val', key3: 1}, true], f: false}
context: {}
result: {key: [1,2,{key2: 'val', key3: 1}, true], f: false}
String Interpolation
The simplest form of substitution occurs within strings, using ${..}
:
template: {message: 'hello ${key}', 'k=${num}': true}
context: {key: 'world', num: 1}
result: {message: 'hello world', 'k=1': true}
The bit inside the ${..}
is an expression, and must evaluate to something
that interpolates obviously into a string (so, a string, number, boolean,).
If it is null, then the expression interpolates into an empty string.
The expression syntax is described in more detail below.
Values interpolate as their JSON literal values:
template: ["number: ${num}", "booleans: ${t} ${f}", "null: ${nil}"]
context: {num: 3, t: true, f: false, nil: null}
result: ["number: 3", "booleans: true false", "null: "]
Note that object keys can be interpolated, too:
template: {"tc_${name}": "${value}"}
context: {name: 'foo', value: 'bar'}
result: {"tc_foo": "bar"}
The string ${
can be escaped as $${
.
Operators
JSON-e defines a bunch of operators. Each is represented as an object with a
property beginning with $
. This object can be buried deeply within the
template. Some operators take additional arguments as properties of the same
object.
$eval
The $eval
operator evaluates the given expression and is replaced with the
result of that evaluation. Unlike with string interpolation, the result need
not be a string, but can be an arbitrary data structure.
template: {config: {$eval: 'settings.staging'}}
context:
settings:
staging:
transactionBackend: mock
production:
transactionBackend: customerdb
result: {config: {transactionBackend: 'mock'}}
The expression syntax is described in more detail below.
Note that $eval
's value must be a string. "Metaprogramming" by providing a
calculated value to eval is not allowed. For example, {$eval: {$eval: "${var1} + ${var2}"}}
is not valid JSON-e.
$json
The $json
operator formats the given value as JSON with sorted keys. It does
not evaluate the value (use $eval
for that). While this can be useful in some
cases, it is an unusual case to include a JSON string in a larger data
structure.
template: {$json: [a, b, {$eval: 'a+b'}, 4]}
context: {a: 1, b: 2}
result: '["a", "b", 3, 4]'
$if
- then
- else
The $if
operator supports conditionals. It evaluates the given value, and
replaces itself with the then
or else
properties. If either property is
omitted, then the expression is omitted from the parent object.
template: {key: {$if: 'cond', then: 1}, k2: 3}
context: {cond: true}
result: {key: 1, k2: 3}
template: {$if: 'x > 5', then: 1, else: -1}
context: {x: 10}
result: 1
template: [1, {$if: 'cond', else: 2}, 3]
context: {cond: false}
result: [1,2,3]
template: {key: {$if: 'cond', then: 2}, other: 3}
context: {cond: false}
result: {other: 3}
$flatten
The $flatten
operator flattens an array of arrays into one array.
template: {$flatten: [[1, 2], [3, 4], [5]]}
context: {}
result: [1, 2, 3, 4, 5]
$flattenDeep
The $flattenDeep
operator deeply flattens an array of arrays into one array.
template: {$flattenDeep: [[1, [2, [3]]]]}
context: {}
result: [1, 2, 3]
$fromNow
The $fromNow
operator is a shorthand for the built-in function fromNow
. It
creates a JSON (ISO 8601) datestamp for a time relative to the current time
(see the now
builtin, below) or, if from
is given, relative to that time.
The offset is specified by a sequence of number/unit pairs in a string. For
example:
template: {$fromNow: '2 days 1 hour'}
context: {}
result: '2017-01-19T16:27:20.974Z'
template: {$fromNow: '1 hour', from: '2017-01-19T16:27:20.974Z'}
context: {}
result: '2017-01-19T17:27:20.974Z'
The available units are day
, hour
, minute
, and second
, for all of which a plural
is also accepted.
$let
The $let
operator evaluates an expression using a context amended with the
given values. It is analogous to the Haskell where
clause.
template: {$let: {ts: 100, foo: 200},
in: [{$eval: "ts+foo"}, {$eval: "ts-foo"}, {$eval: "ts*foo"}]}
context: {}
result: [300, -100, 20000]
The $let
operator here added the ts
and foo
variables to the scope of
the context and accordingly evaluated the in
clause using those variables
to return the correct result.
An expression like {$let: {$eval: "extraVariables"}, in : ..}
is supported. As long as
the value of $let
evaluates to an object with valid key(s), the values of which are
evaluated.
template: {$let: {$if: something == 3, then: {a: 10, b: 10}, else: {a: 20, b: 10}},
in: {$eval: 'a + b'}}
context: {'something': 3}
result: 20
template: {$let: {"b": {$eval: "a + 10"}},
in: {$eval: "a + b"}}
context: {a: 5}
result: 20
template: {$let: {"first_${name}": 1, "second_${name}": 2},
in: {$eval: "first_prize + second_prize"}}
context: {name: "prize"}
result: 3
$map
The $map
operator evaluates an expression for each value of the given array or object,
constructing the result as an array or object of the evaluated values.
When given an array, map always returns an array.
template:
$map: [2, 4, 6]
each(x): {$eval: 'x + a'}
context: {a: 1}
result: [3, 5, 7]
---
template:
$map: [2, 4, 6]
each(x,i): {$eval: 'x + a + i'}
context: {a: 1}
result: [3, 6, 9]
The array or object is the value of the $map
property, and the expression to evaluate
is given by each(var[,key|index])
where var
is the name of the variable containing each
element and key|index
is either the object key or array index of the value. In the case of
iterating over an object and no key|index
var name is given, var
will be an object with
two keys: key
and val
. These keys correspond to a key in the object and its corresponding value.
When $map is given an object, the expression defined by each(var)
must evaluate to an
object for each key/value pair (key
and val
). The objects constructed by each 'each(var)'
can then be merged internally to give the resulting object with later keys overwriting
the previous ones. Otherwise the expression becomes invalid for the $map operator.
template:
$map: {a: 1, b: 2, c: 3}
each(y): {'${y.key}x': {$eval: 'y.val + 1'}}
context: {}
result: {ax: 2, bx: 3, cx: 4}
---
template:
$map: {a: 1, b: 2, c: 3}
each(v,k): {'${k}x': {$eval: 'v + 1'}}
context: {}
result: {ax: 2, bx: 3, cx: 4}
$match
The $match
operator is not dissimilar to pattern matching operators.
It gets an object, in which every key is a string expression(s) to
evaluate to true
or false
based on the context. The result will
be an array of things (all types are supported) that were values
corresponding to the keys that were evaluated to true
. The order of
the things in the array will be arbitrary. If there are no matches,
the result is an empty array.
template: {$match: {"x == 10": "ten", "x == 20": "twenty"}}
context: {x: 10}
result: ["ten"]
template: {$match: {"x == 10 || x == 20": "tens", "x == 10": "ten"}}
context: {x: 10}
one possible result: ["tens", "ten"]
another possible result: ["ten", "tens"]
template: {$match: {"x < 10": "tens"}}
context: {x: 10}
result: []
$merge
The $merge
operator merges an array of objects, returning a single object
that combines all of the objects in the array, where the right-side objects
overwrite the values of the left-side ones.
template: {$merge: [{a: 1, b: 1}, {b: 2, c: 3}, {d: 4}]}
context: {}
result: {a: 1, b: 2, c: 3, d: 4}
$mergeDeep
The $mergeDeep
operator is like $merge
, but it recurses into objects to
combine their contents property by property. Arrays are concatenated.
template:
$mergeDeep:
- task:
payload:
command: [a, b]
- task:
extra:
foo: bar
- task:
payload:
command: [c]
context: {}
result:
task:
extra:
foo: bar
payload:
command: [a, b, c]
$sort
The $sort
operator sorts the given array. It takes a by(var)
property which
should evaluate to a comparable value for each element. The by(var)
property
defaults to the identity function.
template:
$sort: [{a: 2}, {a: 1, b: []}, {a: 3}]
by(x): 'x.a'
context: {}
result: [{a: 1, b: []}, {a: 2}, {a: 3}]
$reverse
The $reverse
operator simply reverses the given array.
template: {$reverse: [3, 4, 1, 2]}
context: {}
result: [2, 1, 4, 3]
Escaping operators
All property names starting with $
are reserved for JSON-e.
You can use $$
to escape such properties:
template: {$$reverse: [3, 2, {$$eval: '2 - 1'}, 0]}
context: {}
result: {$reverse: [3, 2, {$eval: '2 - 1'}, 0]}
Truthiness
Many values can be evaluated in context where booleans are required, not just booleans themselves. JSON-e defines the following values as false. Anything else will be true.
template: {$if: 'a || b || c || d || e || f', then: "uh oh", else: "falsy" }
context: {a: null, b: [], c: {}, d: "", e: 0, f: false}
result: "falsy"
Expression Syntax
Expression are given in a simple Python- or JavaScript-like expression language. Its data types are limited to JSON types plus function objects.
Literals
Literals are similar to those for JSON. Numeric literals only accept integer
and decimal notation. Strings do not support any kind of escaping. The use of
\n
and \t
in the example below depends on the YAML parser to expand the
escapes.
template:
- {$eval: "1.3"}
- {$eval: "'abc'"}
- {$eval: '"abc"'}
- {$eval: "'\n\t'"}
context: {}
result:
- 1.3
- "abc"
- "abc"
- "\n\t"
Array and object literals also look much like JSON, with bare identifiers allowed as keys like in Javascript:
template:
- {$eval: '[1, 2, "three"]'}
- {$eval: '{foo: 1, "bar": 2}'}
context: {}
result:
- [1, 2, "three"]
- {"foo": 1, "bar": 2}
Context References
Bare identifiers refer to items from the context or to built-ins (described below).
template: {$eval: '[x, z, x+z]'}
context: {x: 'quick', z: 'sort'}
result: ['quick', 'sort', 'quicksort']
Arithmetic Operations
The usual arithmetic operators are all defined, with typical associativity and precedence:
template:
- {$eval: 'x + z'}
- {$eval: 's + t'}
- {$eval: 'z - x'}
- {$eval: 'x * z'}
- {$eval: 'z / x'}
- {$eval: 'z ** 2'}
- {$eval: '(z / x) ** 2'}
context: {x: 10, z: 20, s: "face", t: "plant"}
result:
- 30
- "faceplant"
- 10
- 200
- 2
- 400
- 4
Note that strings can be concatenated with +
, but none of the other operators
apply.
Comparison Operations
Comparisons work as expected. Equality is "deep" in the sense of doing comparisons of the contents of data structures.
template:
- {$eval: 'x < z'}
- {$eval: 'x <= z'}
- {$eval: 'x > z'}
- {$eval: 'x >= z'}
- {$eval: 'deep == [1, [3, {a: 5}]]'}
- {$eval: 'deep != [1, [3, {a: 5}]]'}
context: {x: -10, z: 10, deep: [1, [3, {a: 5}]]}
result: [true, true, false, false, true, false]
Boolean Operations
Boolean operations use C- and Javascript-style symbols ||
, &&
, and !
:
template: {$eval: '!(false || false) && true'}
context: {}
result: true
Json-e supports short-circuit evaluation, so if in ||
left operand is true
returning value will be true no matter what right operand is:
context: {}
template: {$eval: "true || b"}
result: true
And if in &&
left operand is false returning value will be false no matter
what right operand is:
context: {}
template: {$eval: "false && b"}
result: false
Object Property Access
Like Javascript, object properties can be accessed either with array-index
syntax or with dot syntax. Unlike Javascript, obj.prop
is an error if obj
does not have prop
, while obj['prop']
will evaluate to null
.
template: {$eval: 'v.a + v["b"]'}
context: {v: {a: 'apple', b: 'bananna', c: 'carrot'}}
result: 'applebananna'
Indexing and Slicing
Strings and arrays can be indexed and sliced using a Python-like indexing scheme. Negative indexes are counted from the end of the value. Slices are treated as "half-open", meaning that the result contains the first index and does not contain the second index. A "backward" slice with the start index greater than the end index is treated as empty.
template:
- {$eval: '[array[1], string[1]]'}
- {$eval: '[array[1:4], string[1:4]]'}
- {$eval: '[array[2:], string[2:]]'}
- {$eval: '[array[:2], string[:2]]'}
- {$eval: '[array[4:2], string[4:2]]'}
- {$eval: '[array[-2], string[-2]]'}
- {$eval: '[array[-2:], string[-2:]]'}
- {$eval: '[array[:-3], string[:-3]]'}
context: {array: ['a', 'b', 'c', 'd', 'e'], string: 'abcde'}
result:
- ['b', 'b']
- [['b', 'c', 'd'], 'bcd']
- [['c', 'd', 'e'], 'cde']
- [['a', 'b'], 'ab']
- [[], '']
- ['d', 'd']
- [['d', 'e'], 'de']
- [['a', 'b'], 'ab']
Containment Operation
The in
keyword can be used to check for containment: a property in an object,
an element in an array, or a substring in a string.
template:
- {$eval: '"foo" in {foo: 1, bar: 2}'}
- {$eval: '"foo" in ["foo", "bar"]'}
- {$eval: '"foo" in "foobar"'}
context: {}
result: [true, true, true]
Function Invocation
Function calls are made with the usual fn(arg1, arg2)
syntax. Functions are
not JSON data, so they cannot be created in JSON-e, but they can be provided as
built-ins or supplied in the context and called from JSON-e.
Built-In Functions and Variables
The expression language provides a laundry-list of built-in functions/variables. Library users can easily add additional functions/variables, or override the built-ins, as part of the context.
Time
The built-in context value now
is set to the current time at the start of
evaluation of the template, and used as the default "from" value for $fromNow
and the built-in fromNow()
.
template:
- {$eval: 'now'}
- {$eval: 'fromNow("1 minute")'}
- {$eval: 'fromNow("1 minute", "2017-01-19T16:27:20.974Z")'}
context: {}
result:
- '2017-01-19T16:27:20.974Z',
- '2017-01-19T16:28:20.974Z',
- '2017-01-19T16:28:20.974Z',
Math
template:
# the smallest of the arguments
- {$eval: 'min(1, 3, 5)'}
# the largest of the arguments
- {$eval: 'max(2, 4, 6)'}
# mathematical functions
- {$eval: 'sqrt(16)'}
- {$eval: 'ceil(0.3)'}
- {$eval: 'floor(0.3)'}
- {$eval: 'abs(-0.3)'}
context: {}
result:
- 1
- 6
- 4
- 1
- 0
- 0.3
Strings
template:
# convert string case
- {$eval: 'lowercase("Fools!")'}
- {$eval: 'uppercase("Fools!")'}
# convert string, number, boolean, or array to string
- {$eval: 'str(130)'}
# strip whitespace from left, right, or both ends of a string
- {$eval: 'lstrip(" room ")'}
- {$eval: 'rstrip(" room ")'}
- {$eval: 'strip(" room ")'}
context: {}
result:
- "fools!"
- "FOOLS!"
- "130"
- "room "
- " room"
- room
Type
The typeof()
built-in returns the type of an object. Its behavior around
null
is reminiscent of JavaScript.
template:
- "${typeof('abc')}"
- "${typeof(42)}"
- "${typeof(42.0)}"
- "${typeof(true)}"
- "${typeof([])}"
- "${typeof({})}"
- "${typeof(typeof)}"
- {$eval: "typeof(null)"}
- "${typeof(null)}"
context: {}
result:
- string
- number
- number
- boolean
- array
- object
- function
- null
- '' # .. which interpolates to an empty string
Length
The len()
built-in returns the length of a string or array.
template: {$eval: 'len([1, 2, 3])'}
context: {}
result: 3