leadpony / JSON-Schema-Test-Suite

A language agnostic test suite for the JSON Schema specifications

JSON Schema Test Suite

This repository contains a set of JSON objects that implementors of JSON Schema validation libraries can use to test their validators.

It is meant to be language agnostic and should require only a JSON parser.

The conversion of the JSON objects into tests within your test framework of choice is still the job of the validator implementor.

Structure of a Test

If you're going to use this suite, you need to know how tests are laid out. The tests are contained in the tests directory at the root of this repository.

Inside that directory is a subdirectory for each draft or version of the schema.

If you look inside the draft directory, there are a number of .json files, which logically group a set of test cases together. Often the grouping is by property under test, but not always, especially within optional test files (discussed below).

Inside each .json file is a single array containing objects. It's easiest to illustrate the structure of these with an example:

    {
        "description": "the description of the test case",
        "schema": {"the schema that should" : "be validated against"},
        "tests": [
            {
                "description": "a specific test of a valid instance",
                "data": "the instance",
                "valid": true
            },
            {
                "description": "another specific test this time, invalid",
                "data": 15,
                "valid": false
            }
        ]
    }

So a description, a schema, and some tests, where tests is an array containing one or more objects with descriptions, data, and a boolean indicating whether they should be valid or invalid.

Coverage

Drafts 07, 06, 04 and 03 should have full coverage, with drafts 06 and 07 being considered current and actively supported.

Draft 2019-09 support is under development. Contributions are very welcome, especially from implementers as they add support to their own implementations.

Bug fixes will be made as needed for draft-04 as it is still the most widely used, while draft-03 is long since deprecated.

If you see anything missing from the current supported drafts, or incorrect on any draft still accepting bug fixes, please file an issue or submit a PR.

Who Uses the Test Suite

This suite is being used by:

Clojure

• jinx
• json-schema

Coffeescript

• jsck

C++

• Modern C++ JSON schema validator

Dart

• json_schema

Elixir

• ex_json_schema

Erlang

• jesse

Go

• gojsonschema
• validate-json

Haskell

• aeson-schema
• hjsonschema

Java

• json-schema-validator
• everit-org/json-schema
• networknt/json-schema-validator
• Justify

JavaScript

• json-schema-benchmark
• direct-schema
• is-my-json-valid
• jassi
• JaySchema
• json-schema-valid
• Jsonary
• jsonschema
• request-validator
• skeemas
• tv4
• z-schema
• jsen
• ajv
• djv

Node.js

For node.js developers, the suite is also available as an npm package.

Node-specific support is maintained in a separate repository which also welcomes your contributions!

.NET

• Newtonsoft.Json.Schema
• Manatee.Json

PHP

• json-schema
• json-guard

PostgreSQL

• postgres-json-schema
• is_jsonb_valid

Python

• jsonschema
• fastjsonschema
• hypothesis-jsonschema

Ruby

• json-schema
• json_schemer

Rust

• valico

Swift

• JSONSchema

If you use it as well, please fork and send a pull request adding yourself to the list :).

Contributing

If you see something missing or incorrect, a pull request is most welcome!

There are some sanity checks in place for testing the test suite. You can run them with bin/jsonschema_suite check && npm test or tox && npm test. They will be run automatically by Travis CI as well.