cjoudrey/graphql-cats

GraphQL Compatibility Acceptance Tests

The aim of this project is to provide a set of compatibility acceptance tests for libraries that implement the GraphQL specification. Since GraphQL server implementations are written in different programming languages, this project defines test cases in a language independent format (YAML).

Using this test suite has following advantages:

The users and the author of a GraphQL library can be more confident that a library is compliant to the GraphQL specification and if not, which parts of the library are not compliant
It reduces amount of work a library implementor needs to do
Makes it much easier for existing implementations to keep up with the specification changes

GraphQL implementations that have already implemented or currently working on the implementation of graphql-cats driver:

If you are working on the driver for graphql-cats for your project, we would appreciate if you could create a PR and add it into this list.

Contribution & Open Questions

At the moment this project finds itself in an infant phase. Any contributions are very welcome! Please join us in a gitter chat. If you would like to join this effort and be part of the project, please let us know.

There are still a number of open questions that we need to figure out together:

Repository format

scenarios folder contains a set of *.yaml and *.graphql files. Every subfolder represents a particular part of GraphQL specification. *.yaml files contain scenarios with a list of test cases. *.graphql files contain SDL schema definitions that are used in some of the scenarios (they are explicitly referenced).

Scenario File Format

Every scenario is a YAML file with following structure:

scenario - String - the name of this scenario
background - Object (optional) - common definitions used by all of the tests
- schema - String (optional) - inline GraphQL SDL schema definition
- schema-file - String (optional) - SDL schema definition file path relative to the scenario file
- test-data - Object (optional) - test data used for query execution and directives
- test-data-file - String (optional) - test data file path relative to the scenario file. File can be in either JSON or YAML format.
tests - Array of Objects - list of tests
- name - String - a name of the test
- given - Object - input information for the test
  - query - String - the GraphQL query to execute an action against
  - schema - String (optional) - inline GraphQL SDL schema definition
  - schema-file - String (optional) - SDL schema definition file path relative to the scenario file
  - test-data - Object (optional) - test data used for query execution and directives
  - test-data-file - String (optional) - test data file path relative to the scenario file. File can be in either JSON or YAML format.
- when - Object - action that should be performed in the test. See the Actions section for a list of available actions.
- then - Object | Arrays of Objects - assertions that verify result of an action. See the Assertions section for a list of available actions.

Definitions in the given part of a test may override definitions defined in the background section.

Actions

Query parsing
- parse - Boolean - just parses the query
Query validation
- validate - Array of Strings - the list of validation rule names to validate a query against. This action will only validate query without executing it.
Query execution
- execute - Boolean | Object - executes a query
  - operation-name - String (optional) - the name of an operation to execute (in case query contains more than one)
  - variables - Object (optional) - variables for query execution
  - validate-query - Boolean (optional) - true if query should be validated during the execution, false otherwise (true by default)
  - test-value - String (optional) - the name of a field defined in the test-data. This value should be passed as a root value to an executor.

Assertions

Validation/Parsing is successful
- passes - Boolean - verifies that validation was successful. Only applicable in conjunction with query validation/parsing action
Parsing syntax error
- syntax-error - Boolean - query contains a syntax error. Only applicable in conjunction with query parsing action (No text matching takes place, it just verifies the fact that there is a syntax error)
Data match
- data - Object - compares the data object with the result of a query execution. Only applicable in conjunction with query execution action
Error count
- error-count - Number - number of the errors in execution/validation results
Error code match
- error-code - String - execution/validation results contains an error with provided code. Each error code must be described in the error mapping. See Error Mapping section for more details.
- args - Object (optional) - arguments for an error code (might be used as the placeholders in a human-readable error message representation)
- loc - Array of Objects | Array of Arrays of Numbers (optional) - a list of error locations
  - line - Number
  - column - Number
Error contains match
- error - String - execution/validation results contain provided error message (provided error message may contain only part of the actual message)
- loc - Array of Objects | Array of Arrays of Numbers (optional) - a list of error locations
  - line - Number
  - column - Number
Error regex match
- error-regex - String - execution/validation results contain provided error message (uses provided regular expressions to match an error message)
- loc - Array of Objects | Array of Arrays of Numbers (optional) - a list of error locations
  - line - Number
  - column - Number
Execution exception contains match
- exception - String - execution may throw an exception during the execution (for instance, if operation name is not provided, but query contains more than one named operation). This assertion verifies the message of this exception (provided error message may contain only part of the actual message). Only applicable in conjunction with query execution action
Execution exception regex match
- error-regex - String - execution may throw an exception during the execution (for instance, if operation name is not provided, but query contains more than one named operation). This assertion verifies the message of this exception (uses provided regular expressions to match an error message). Only applicable in conjunction with query execution action

Error Mapping

Error mapping is located in scenarios/error-mapping.yaml. It is a YAML file which defines a mapping between an error code and the human-readable message which can be seen in the reference implementation for this code. It also provides additional meta-information for every individual error code, like links to the specification and reference implementation which define these errors.

Every error code has following structure in the mapping:

message - String - human-readable message which can be seen in the reference implementation for this code. If error code takes arguments, then arguments might be used in the message with as the placeholders (placeholder syntax: ${argumentName}). For example (fieldName and type are the arguments): Field "${fieldName}" must not have a selection since type "${type}" has no subfields.
references
- spec - String - link to a place in the specification which defines this error.
- implementation - String - link to a place in the reference implementation which defines this error.

Execution Semantics

All test cases and scenarios are self-contained. This means that they contain the schema definition, query and test data for an execution. Schema definition expected to be materialized in executable form. By default, fields' resolve function must return the values provided with the test-data/test-data-file and test-value properties which are defined in a scenario file. Some fields may have special behaviour which is defined via schema directives which you can find in the next section.

Schema Directives

@resolveString

directive @resolveString(value: String!) on FIELD_DEFINITION

Resolves String field with provided value. Value may contain a placeholders which refer to the field arguments and must be replaced during the field resolution.

Example:

for given schema definition:

type Query {
  article(title: String): String @resolveString(value: "Test article with title '$title'")
}

schema {
  query: Query
}

and query:

{
  article(title: Foo)    
}

result of an execution should produce following JSON:

{
  "data": {
    "article": "Test article with title 'Foo'"
  }
}

@argumentsJson

directive @argumentsJson on FIELD_DEFINITION

Resolves String field with compact JSON representation of all field arguments.

@resolvePromiseString

directive @resolvePromiseString(value: String!) on FIELD_DEFINITION

Resolves String field with provided value which may contain argument placeholders. Internally field should be resolved with successful promise or an equivalent of promise. Ideally short delay should be applied before actual resolution. If implementation does not support promises, then it may return an eager value.

@resolveEmptyObject

directive @resolveEmptyObject on FIELD_DEFINITION

Resolves ObjectType field with an empty object.

@resolveTestData

directive @resolveTestData(name: String!) on FIELD_DEFINITION

Resolves field with a value provided via test-data property.

@resolvePromiseTestData

directive @resolvePromiseTestData(name: String!) on FIELD_DEFINITION

Resolves field with a value provided via test-data property. Internally field should be resolved with successful promise or an equivalent of promise. Ideally short delay should be applied before actual resolution. If implementation does not support promises, then it may return an eager value.

@resolvePromise

directive @resolvePromise on FIELD_DEFINITION

Resolves field with context value (as in the default case). Internally field should be resolved with successful promise or an equivalent of promise. Ideally short delay should be applied before actual resolution. If implementation does not support promises, then it may return an eager value.

@resolveError

directive @resolveError(message: String!) on FIELD_DEFINITION

Field resolution fails with provided error message.

@resolveErrorList

directive @resolveErrorList(values: [String!]!, messages: [String!]!) on FIELD_DEFINITION

Field of type [String] fails with provided errors messages. Even though field resolution fails, it still must produce a value which contains provided values.

@resolvePromiseReject

directive @resolvePromiseReject(message: String!) on FIELD_DEFINITION

Field resolution fails with provided error message. Internally field should be resolved with rejected promise or an equivalent of promise. Ideally short delay should be applied before actual resolution. If implementation does not support promises, then it may return an eager error.

@resolvePromiseRejectList

directive @resolvePromiseRejectList(values: [String!]!, messages: [String!]!) on FIELD_DEFINITION

Field of type [String] fails with provided errors messages. Even though field resolution fails, it still must produce a value which contains provided values. Internally field should be resolved with rejected promise or an equivalent of promise. Ideally short delay should be applied before actual resolution. If implementation does not support promises, then it may return an eager error.

Test Data

Test data is a JSON/YAML object where every top-level field may be referenced by name within a scenario (with test-value property), schema directive (@resolveTestData, @resolvePromiseTestData) or test data itself (with {"$ref": "name"} value).

Let's look at small scenario example that takes advantage of the test data and references:

scenario: "Execute: Handles basic execution tasks"
tests:
  - name: test scenario
    given:
      schema: |
        type Basket {
          name: String
          content: [Fruit]
        }

        type Fruit {
          name: String
          inBasket: Basket
        }

        type Query {
          apples: Basket @resolveTestData(name: "tasty")
        }
      test-data:
        green-apple: {name: "Green Apple!", inBasket: {$ref: tasty}}
        tasty:
          name: Tasty fuits!
          content: [{name: "Red Apple!", inBasket: {$ref: tasty}}, {$ref: green-apple}]
      query: |
        query {
          apples {
            content {
              name
              inBasket {
                name
              }
            }
          }
        }
    when:
      execute: true
    then:
      data:
        apples:
          content:
          - name: Red Apple!
            inBasket:
              name: Tasty fuits!
          - name: Green Apple!
            inBasket:
              name: Tasty fuits!

The test defines 2 data elements: "green-apple" and "tasty". These JSON objects are then referenced by name from the @resolveTestData(name: "tasty") directive and withing the test data itself.

Example Driver Implementation

In order to use this test suite in your GraphQL library, you need to implement a small driver that reads scenario files and executes the tests. An example driver can be found in Sangria (scala GraphQL implementation):

CatsSupport.scala

Main entry point that generates the tests can be found in the generateTests function.

In general, driver executes following steps:

Load scenario data from the YAML file (CatsScenarioData)
Generate an executable schema based on the SDL provided in the scenario ("given" part of the scenario) (CatsScenarioExecutor)
Execute a test query against the generated schema ("when" part of the scenario) (CatsScenarioExecutor.executeAction)
Assert results of the execution ("then" part of the scenario) (CatsAssertions)

cjoudrey / graphql-cats