kylef / JSON-Schema-Test-Suite

A language agnostic test suite for the JSON Schema specifications

JSON Schema Test Suite

This repository contains a set of JSON objects that implementors of JSON Schema validation libraries can use to test their validators.

It is meant to be language agnostic and should require only a JSON parser.

The conversion of the JSON objects into tests within your test framework of choice is still the job of the validator implementor.

Structure of a Test

The tests in this suite are contained in the tests directory at the root of this repository. Inside that directory is a subdirectory for each draft or version of the specification.

Inside each draft directory, there are a number of .json files and one or more special subdirectories. The subdirectories contain .json files meant for a specific testing purpose, and each .json file logically groups a set of test cases together. Often the grouping is by property under test, but not always.

The subdirectories are described in the next section.

Inside each .json file is a single array containing objects. It's easiest to illustrate the structure of these with an example:

{
    "description": "The description of the test case",
    "schema": {
        "description": "The schema against which the data in each test is validated",
        "type": "string"
    },
    "tests": [
        {
            "description": "Test for a valid instance",
            "data": "the instance to validate",
            "valid": true
        },
        {
            "description": "Test for an invalid instance",
            "data": 15,
            "valid": false
        }
    ]
}

In short: a description, a schema under test, and some tests, where each test in the tests array is an objects with a description of the case itself, the instance under test, and a boolean indicating whether it should be valid or invalid.

Test Subdirectories

There is currently only one subdirectory that may exist within each draft directory. This is:

1. optional/: Contains tests that are considered optional.

Coverage

Drafts 07, 06, 04, and 03 should have full coverage, with tests for drafts 06, 07, 2019-09 and 2020-12 being considered current and actively supported. Drafts 2019-09 and 2020-12 are almost fully covered.

Contributions are very welcome, especially from implementers as they add support to their own implementations.

If you see anything missing from the current supported drafts, or incorrect on any draft still accepting bug fixes, please file an issue or submit a PR.

Who Uses the Test Suite

This suite is being used by:

Clojure

• jinx
• json-schema

Coffeescript

• jsck

Common Lisp

• json-schema

C++

• Modern C++ JSON schema validator

Dart

• json_schema

Elixir

• ex_json_schema

Erlang

• jesse

Go

• gojsonschema
• validate-json

Haskell

• aeson-schema
• hjsonschema

Java

• json-schema-validator
• everit-org/json-schema
• networknt/json-schema-validator
• Justify
• Snow
• jsonschemafriend

JavaScript

• json-schema-benchmark
• direct-schema
• is-my-json-valid
• jassi
• JaySchema
• json-schema-valid
• Jsonary
• jsonschema
• request-validator
• skeemas
• tv4
• z-schema
• jsen
• ajv
• djv

Node.js

For node.js developers, the suite is also available as an npm package.

Node-specific support is maintained in a separate repository which also welcomes your contributions!

.NET

• Newtonsoft.Json.Schema
• Manatee.Json

Perl

• JSON::Schema::Draft201909
• JSON::Schema::Tiny
• Test::JSON::Schema::Acceptance

PHP

• opis/json-schema
• json-schema
• json-guard

PostgreSQL

• postgres-json-schema
• is_jsonb_valid

Python

• jsonschema
• fastjsonschema
• hypothesis-jsonschema
• jschon

Ruby

• json-schema
• json_schemer

Rust

• jsonschema
• valico

Swift

• JSONSchema

If you use it as well, please fork and send a pull request adding yourself to the list :).

Contributing

If you see something missing or incorrect, a pull request is most welcome!

There are some sanity checks in place for testing the test suite. You can run them with bin/jsonschema_suite check or tox. They will be run automatically by GitHub Actions as well.