SWG needs to consider the nuts and bolt of the testability of the API
pvretano opened this issue · comments
The SWG needs to discuss and consider the testability of the requirements of the specification. Specifically, in order to test certain requirements of the specification (e.g. jobs conformance class) the processes being tested needs to have certain characteristics. For example, the process has to run asynchronously for a sufficient time to allow the test engine to test all the intermediate status messages.
So the question is, how do we convey, describe these preconditions for testability?
With regards to the jobs conformance class, some ideas discussed during the TB17 CITE kickoff included:
- Require that the server being tested implement an "Echo" process. This process could include a pause argument to simulate long running times for job creating and execution.
- Have the client submit an execute request to any process it deploys. The test engine can parse that execute request and figure out if, for example, the process is being run asynchronously and then decide to test the intermediate status messages.
In all this discussion, it was emphasised that the test suite is NOT testing the output of the process. That is irrelevant. Only the interface behaviour should be tested.
Although I have concentrated on the jobs conformance class in the description of this issue, the issue is a general one and the SWG needs to consider in the ATS how all the requirement might actually be tested in the real world.
- Require that the server provides one or more example values for the process input as part of the JSON Schema that describes the input. That would have the added benefit that this would also help other clients or developers to get started with a process by having an example that works.
@cportele I would not want to make it a requirement ... is a recommendation sufficient?
@pvretano - In general, I agree that this is something for a recommendation, but there are aspects why it at least makes sense to consider making it a requirement:
The general idea is that the schema is already detailed enough so that clients can create valid and meaningful requests. This is similar to the queryables from Features where the client has to be able to understand the value range of a property to create meaningful filter predicates. However, it will not be possible to express all value constraints as schema constraints. For a user / developer the additional information can then be included in the description member, but this does not help clients, in particular test engines, to interact with an API without human intervention. If we also want to support those, and the answer may well be "no", then this would require that sufficient information is provided by the API itself to create valid requests. Examples seems to be the easiest way to achieve this. It is also in a way consistent with the general discussion how important examples are; they are not only important in the context of standards to make them more understandable, the same is valid for implementations.
A variant of the example approach could also be to require meaningful default values for all inputs. This is similar to what we do in OGC API standards with query parameters. In most of them, no query parameter is required, there is always a default value, so clients can send a GET request to resource and can expect to get a success response back and not a client error response.
In fact, JSON Schema also points out that aspect of default values (link):
Implementations MAY use the value(s) of "default", if present, as an additional example. If "examples" is absent, "default" MAY still be used in this manner.
In that approach, an empty job request should always result in a successful execution.
This is also connected to #119
I am reopening this because the recommendation 14 in the specification is a little weak. If we say that the Echo process is one option for testability then I think we need to provide a formal description of that process that everyone needs to implement.
Here is a simplified version of the Echo process for the specification that I whipped up as a starting point for a standard process description for the Echo process. It remove the more esoteric input types in the example. The idea is to has an as-simple-as-possible echo process that still tests a good chunk of the core.
If the processes is executed asynchronously, then implementations should introduce a delay in the background processes to give the CITE test tool a change to test polling, etc.
Comments?
{
"id": "EchoProcess",
"title": "Echo Process",
"description": "This process accepts and number of input and simply echoes each input as an output.",
"version": "1.0.0",
"jobControlOptions": [
"async-execute",
"sync-execute"
],
"outputTransmission": [
"value",
"reference"
],
"inputs": {
"stringInput": {
"title": "String Literal Input Example",
"description": "This is an example of a STRING literal input.",
"schema": {
"type": "string",
"enum": [
"Value1",
"Value2",
"Value3"
]
}
},
"measureInput": {
"title": "Numerical Value with UOM Example",
"description": "This is an example of a NUMERIC literal with an associated unit of measure.",
"schema": {
"type": "object",
"required": [
"value",
"uom"
],
"properties": {
"measurement": {
"type": "number"
},
"uom": {
"type": "string"
},
"reference": {
"type": "string",
"format": "uri"
}
}
}
},
"dateInput": {
"title": "Date Literal Input Example",
"description": "This is an example of a DATE literal input.",
"schema": {
"type": "string",
"format": "dateTime"
}
},
"doubleInput": {
"title": "Bounded Double Literal Input Example",
"description": "This is an example of a DOUBLE literal input that is bounded between a value greater than 0 and 10. The default value is 5.",
"schema": {
"type": "number",
"format": "double",
"minimum": 0,
"maximum": 10,
"default": 5,
"exclusiveMinimum": true
}
},
"arrayInput": {
"title": "Array Input Example",
"description": "This is an example of a single process input that is an array of values. In this case, the input array would be interpreted as a single value and not as individual inputs.",
"schema": {
"type": "array",
"minItems": 2,
"maxItems": 10,
"items": {
"type": "integer"
}
}
},
"complexObjectInput": {
"title": "Complex Object Input Example",
"description": "This is an example of a complex object input.",
"schema": {
"type": "object",
"required": [
"property1",
"property5"
],
"properties": {
"property1": {
"type": "string"
},
"property2": {
"type": "string",
"format": "uri"
},
"property3": {
"type": "number"
},
"property4": {
"type": "string",
"format": "dateTime"
},
"property5": {
"type": "boolean"
}
}
}
}
},
"outputs": {
"stringOutput": {
"schema": {
"type": "string",
"enum": [
"Value1",
"Value2",
"Value3"
]
}
},
"measureOutput": {
"schema": {
"type": "object",
"required": [
"value",
"uom"
],
"properties": {
"measurement": {
"type": "number"
},
"uom": {
"type": "string"
},
"reference": {
"type": "string",
"format": "uri"
}
}
}
},
"dateOutput": {
"schema": {
"type": "string",
"format": "dateTime"
}
},
"doubleOutput": {
"schema": {
"type": "number",
"format": "double",
"minimum": 0,
"maximum": 10,
"default": 5,
"exclusiveMinimum": true
}
},
"arrayOutput": {
"schema": {
"type": "array",
"minItems": 2,
"maxItems": 10,
"items": {
"type": "integer"
}
}
},
"complexObjectOutput": {
"schema": {
"type": "object",
"required": [
"property1",
"property5"
],
"properties": {
"property1": {
"type": "string"
},
"property2": {
"type": "string",
"format": "uri"
},
"property3": {
"type": "number"
},
"property4": {
"type": "string",
"format": "dateTime"
},
"property5": {
"type": "boolean"
}
}
}
}
},
"links": [
{
"href": "https://processing.example.org/oapi-p/processes/EchoProcess/execution",
"rel": "http://www.opengis.net/def/rel/ogc/1.0/execute",
"title": "Execute endpoint"
}
]
}
SWG meeting from 2024-03-18: For version 1.1: remove reference to echo process. ETS API - Processes: make it work with execute examples in openapi descriptions of services under test.