This document covers standards for the development of REST APIs. It is intended as a guide to designing the interface of a service.
This document includes standard approaches for
- URI Schema
- JSON Response
- Error handling
- Status Codes
- Versioning
- Authentication
This document does not define an implementation or technology to be used.
- JSON API - org
- Jeff Bezo's mandate at Amazon
- JSON Web Token - http://jwt.io/
- Versioning discussion - http://www.troyhunt.com/2014/02/your-api-versioning-is-wrong-which-is.html
- More on versioning - http://stackoverflow.com/questions/10742594/versioning-rest-api and http://barelyenough.org/blog/tag/rest-versioning/
The URI schema provides an identity for a resource. The identity of the resource MUST NOT change over time. The resource MUST always reference the same thing. If I have a https://domain.com/users/1 and get a name of John Smith. If I go back to the same resource a week later, it should still reference John Smith.
A resource URI can also identify a collection of like resources. https://domain.com/users provides a collection of all available users.
When accessing a collection of entities in a REST API, you'll ALWAYS have a resource, which SHOULD BE a noun and sometimes you MAY have modifiers.
A URI within a REST API is going to be made up of several components.
- Protocol
- Domain
- Namespace
- Resource
- Header
- Identifier
- Modifier
- Relationship
- Verb
- Protocol
- Domain Name
- Resource
- Namespace
- Identifier
- _Verb
- Relationship
- Modifier
The protocol SHOULD BE HTTPS using (SSL / TLS, etc.)
In order to maintain security of the resources, a secure connection between the client and the server SHOULD BE established to prevent unauthorized sniffing of the contents.
This is even more important if the service is exposed to the outside. This will typically require clients to install aX.509 Digital Certificate is required on at least one end of the connection. The Digital Certificate is usually installed at the Server end because it makes it simple for any end user to make a secure SSL or TLS connection to the server without a Digital Certificate on the client end.
In addition to the secure connection, some form of authentication SHOULD BE implemented. See Authentication for more information on the authentication options.
- The domain is the domain of the service provider providing the services.
- This should remain constant for a given set of resources.
- It MUST NOT include a port number.
- It MUST NOT be a physical name of any physical server.
- It SHOULD BE be a logical name that represents the business domain these API services expose.
- It SHOULD identify that resources served from this domain are an API - rather than a web site. E.g. https://api.werner.com
A namespace SHOULD be provided to allow for multiple key concepts to be provided on the same domain.
A namespace provides a mechanism to logically partition groups of resources within a single domain. In particular, it can help prevent resource name collisions across broad general concepts. Some namespace examples are /quote, /pricing, /carrier. It is possible that some resource names may be shared across these spaces, such as /alerts. Using namespaces allows us to have /carrier/alerts and /pricing/alerts and treat them as different resources within their context.
The namespace SHOULD BE a singular noun. In the case of some words like /news - that is the noun.
If the domain does not adequately separate the logical business domains and data sets, than a namespace MUST BE used.
A resource is an abstract set of information.
Our standard for new REST API development requires all resources to be pluralized nouns.
The resource component of the URI is the resource type being requested. Examples might be "users", "customers" or "products". Our standard requires the resource in a URI to be pluralized. So user becomes users, customer becomes customers and so on.
The identifier component of the URI is a unique identifier for the resource being requested.
It is important that the identifier is an identifier for the resource and NOT the identifier of a relationship.
The identifier can be any type:
- Number
- String - watch out for special URL characters in string based identifiers such as $ & + , / : ; = ? @ - See: RFC3986 and RFC1738
- UUID
- Composite
If a resource has multiple (Composite) identifiers to represent a unique instance, then multiple path variables can be set. E.g. If a resource is stock symbol, made up of an exchange id and a stock symbol than it would be exposed with a URI of /stocksymbol/{exchangeId}/{symbol}. This is a relatively rare case and is used to support legacy database resources. Ideally new resources being created have a single unique identifier.
The modifier component of the URI is used primarily in getting collections of objects (usually acting as a simple filter). Modifiers are allowed on any route, and MUST be passed as part of the query string. Modifiers are typically used to filter a collection, so are most commonly applied to GET requests of resource lists.
A /resource will provide a list of all possible instances for that resource name. So /categories will provide all categories and /users will provide all users. In many cases the list can be small and finite, say less than 50 instances and would not need any direct modifiers as the client can analyzes the list as a whole. In many cases though, the list can be very large - sometimes in the millions of instances. In these cases modifiers are added to restrict the instances based on the criteria provided.
A modifier to a collection does one of the following:
- Restricts the number of instances of the resource returned E.g. /carrier/history?originCity=Dallas&destCity=Omaha&type=premier
- Restricts the set of fields to be supplied in each instance of the resource. E.g./carrier/owners/{OwnerId}?values=name,phone - only the name and phone fields would be returned in the response
When getting a list of users, your URI would be something like: GET /users It would be expected that this resource would return an array of user objects. In some cases, you might want modifiers to filter the result. In that case, your URI might be something like: GET /users?lastName="Smith" It would be expected that this call would return an array of user objects containing only users who have a lastName equal to 'Smith'. There are some common query parameters that SHOULD be used when managing common paging type capabilities
- limit={number} - should indicate the maximum number of records to be provided in the result.
- page={pageNumber} - the page number to be requested during pagination type requests
- sort={sortString} - the sort string can be a comma separated list of sort options based on fieldName_direction. E.g. 'birthDate_desc,firstName_asc'. In essence any parse-able string that is understood by the service.
- expand={expandFields} - see further details below, but it offers the ability to instruct the server to resolve resources on the server side before sending the response.
Relationships help provide contextual relevance to a resource. E.g. Alerts can be associated with many different things. pricing alerts, quotes alerts, carrier alerts. So we might have /carrier/{carrierId}/alerts. This is equivalent to /alerts?carrierIds={commodityId}
The relationship is expressed as a plural noun. Often refers to an associated resource that can be stand alone, but the known identifier is that of the primary resource.
If a modifier query parameter is applied to a resource with a relationship - E.g. /customers/123/orders?orderStatus=cancelled, the modifier is on the collection of orders, not on the collection of customers.
The verb component of the URI is only used in one of the special cases outlined later in this standard. Common verbs are "start", "stop", and "execute".
Verbs SHOULD be prefixed with an underscore (_), so "start" becomes "_start" and "execute" becomes "_execute" and so on. A path to start alert watching an owner might look something like: POST /carrier/owners/{ownerId}/alerts/_start. The reason for the underscore is to make them standout as executions rather than resource manipulation. Verbs SHOULD be prefixed with an underscore, however, it is not a requirement. Stay consistent within an API (meaning if the verb routes currently do not have an underscore, then don't introduce that paradigm mid-development)
To access an individual resource you MUST have a noun and an identifier. You SHOULD NOT use modifiers in a details route (but there are occasionally exceptions).
Your URI would be something like: GET /users/1
When updating a resource, you should use the HTTP verbs to distinguish the action. Do not put a verb in the URI.
When creating a new resource, POST to the resource. E.g. to create a new user use POST /users
When updating an existing resource PUT to the resource identifier. E.g. to update user with id 3 use PUT /users/3 passing in the complete updated object
When deleting a resource, DELETE to the resource. E.g. to delete user with id 4 use DELETE /users/4. Note, in most cases this should still resort to a logical delete in the underlying data store. Requests made with the DELETE method DO NOT return a response body! The success or failure of the request should be determined by the status code of the response.
When doing a partial update of a resource use PATCH. E.g. to update a portion of user with id 1 use PATCH /users/1 (Your request body would only contain the key/value pairs of the fields being updated)
All response documents SHOULD prefer JSON as the data format. In a case where a business requirement dictates otherwise an API SHOULD offer JSON data format as an alternative.
A JSON object MUST be at the root of every request and response containing data. This object defines the documents 'top level.
Response documents MUST contain at least one of the following top-level members:
- data: documents primary data
- errors: an array of error objects
- meta: contains non-standard meta-information
'data' and 'errors' MUST NOT coexist in the same response.
Responses SHOULD be shallow and relate only to the resource being requested. That is, if a customers resource is being requested, the response should only contain the attributes related to customer. Any customer attributes that reference a separate resource should be created as links to the associated resource.
E.g. GET /customers/1
| { "data": { "id": 1, "self": "/customers/1", "attributes": { "firstName": "John", "lastName": "Smith", "dateOfBirth": "05/05/1980" }, "links": [ { "type": "addresses", "ref": "/customers/1/adresses" }, { "type": "orders", "ref": "/customers/1/orders" }, { "type": "accountManager", "ref": "/employees/4564" } ] }} |
|---|
Getting a collection response.
E.g. GET /customers
| { "data": [ { "id": 1, "self": "/customers/1", "attributes": { "firstName": "John", "lastName": "Smith", "dateOfBirth": "05/05/1980" }, "links": [ { "type": "addresses", "ref": "/customers/1/adresses" }, { "type": "orders", "ref": "/customers/1/orders" }, { "type": "accountManager", "ref": "/employees/4564" } ] }, { "id": 2, "self": "/customers/2", "attributes": { "firstName": "Jane", "lastName": "Doe", "dateOfBirth": "06/06/1986" }, "links": [ { "type": "addresses", "ref": "/customers/2/adresses" }, { "type": "orders", "ref": "/customers/2/orders" }, { "type": "accountManager", "ref": "/employees/4564" } ] } ]} |
|---|
The result provides a reference to additional resources if needed, but does not explicitly provide details about them in the request. If details are required, the request may request them to be expanded.
E.g GET /customers/1?expand=accountManager,addresses would return
| { "id": 1, "self": "/customers/1", "attributes": { "firstName": "John", "lastName": "Smith", "dateOfBirth": "05/05/1980" }, "links": [ { "type": "addresses", "ref": "/customers/1/adresses" }, { "type": "orders", "ref": "/customers/1/orders" }, { "type": "accountManager", "ref": "/employees/4564", "data": null } ], "expanded": [ { "type": "addresses", "data": [ { "id": 444, "self": "/addresses/444", "attributes": { "addressLine1": "100 First Street", "stateCode": "MN", "zipCode": "55111" } } ] }, { "type": "accountManager", "data": [ { "id": 4564, "self": "/employees/4564", "attributes": { "firstName": "Paul", "lastName": "Blart" } } ] } ]} |
|---|
If there is no data to return, then the "data" key MUST be an empty array or object No Data "data": [] OR "data": {}
If there was an error, than the response objects error attribute should be populated with information describing the error. A very nice to have is a doc attribute that is a URL reference to an online documentation that provides additional information about the error.
| { "error": { "code": "4564", "messages": [ { "type": "error", "message": "There was an error fetching customer 1", "doc": " } ] }} |
|---|
The "messages" key MUST be an array (either empty, or containing message dictionaries)
Each message MUST be a dictionary containing as least the keys of "type" and "message"
| { "type": "info", "message": "Rainbow Dash is the greatest of the mane six ponies!"} |
|---|
The message type MUST be one of the following:
- info - Informational messages for the user/client such as notifications about successful jobs or general feedback
- error - Error details when an error has occurred
- warning - Warning messages to notify the user/client of things like sub-optimal configuration settings
- debug - Debug messages. MUST NOT ever be included in a PROD API response
All dates being sent over REST APIs MUST be ISO-8601 encoded.
For more information about the ISO-8601 standard format, see https://en.wikipedia.org/wiki/ISO_8601
Should we use RFC3339 ? This is an extension to ISO-8601 required a complete representation of date and time: http://tools.ietf.org/html/rfc3339
Additional data MAY be added at any level of the response envelope, however, all response payload data SHOULD be included under the "data" key. Any data within the response envelope for debugging purposes MUST have an underscore ( _ ) prefix for the key.
When a dictionary contains a NULL value for a key, the key SHOULD still be returned with the NULL value rather than trimming the key/value from the dictionary.
Last but not least, one of the most important things about any REST API is the status code returned with the response. Below are some of the standard status codes we implement in our APIs. Error status codes SHOULD also include an "error" message in the "messages" key of the response envelope.
- 200 - OK
- Returned when everything with the request went as expected and there were no errors.
- 400 - Bad Request
- Returned when the submitted request body did not match expectations. (Used primarily in POST/PUT/PATCH calls)
- Can also be used when required headers were not sent.
- 401 - Unauthorized
- Returned when attempting to access a resource without a valid login.
- 403 - Forbidden
- Returned when attempting to access a resource to which you don't have access (but you are logged in)
- 404 - Not Found
- Returned when the requested resource could not be found
- 423 - Locked
- We use this code when the server running the API is in "Maintenance Mode" meaning it is undergoing a deployment or upgrade.
- 500 - Internal Error
- When no other error code has been manually specified, or an exception has been thrown and not caught, the server will return a 500 error code. Ideally, our APIs should never throw a 500 error.
- 501 - Not Implemented
- Returned when the route being accessed has not yet been implemented.
It is very important that a REST API be versioned. By versioning the API, you provide for the ability to introduce change to the API without inadvertently breaking the API for your clients. A new version should only be introduced if it would be a breaking change. Examples of this is removal of an expected attribute, or a shift in the structure such as changing an attribute value from an object to an array of objects.
This can sometimes be handled by providing both options in the response. E.g. if an attribute value used to be an object, but now has to be an array of objects, a new attribute could be introduced that is the array of objects and the original attribute can simply hold the first object. This approach MAY BE used, it works, but it does 'dirty' the model. This can often be introduced as a temporary work around until all clients have modified their code to look at the array based portion of the model first and then 'cleaning' the model to remove the old single object.
A better approach is to provide a version number for the API. There is much debate as to whether this version number should exist in the URL or in the header. Our choice for this standard is to keep it in the header. The reason for this is that the URI represents the identity of the resource. By having a version in the URL, a new version number effectively changes the identity of a resource which is essentially still the same resource, and only the representation has changed. See: Your API versioning is wrong and http://stackoverflow.com/questions/10742594/versioning-rest-api and http://barelyenough.org/blog/tag/rest-versioning/
By providing a version number in a custom header named WernerVersion the URL representing the identity of the resource does not change. The prefix of Werner was selected to prevent any future collisions that may occur if the HTTP spec changes. E.g.
Header: Version: 2
This is a simpler implementation and avoids overloading the Accept Header than putting the version number in the Accept Header like this: application/vnd.news.v2+json - this single header is identifying the entity type, the version and the format.
If no version number is provided in the header, ALWAYS respond with the earliest supported version. Most often this will be version 1. This MUST be the default as we do not want to break the contract and the client by supplying the representation of the latest version. The latest is unstable to a client expecting the representation of the version they have been using.
If a client requests a resource with a version number in the header that is not supported for that version number, than return a 404 Bad Request.
The response to the client SHOULD indicate the latest version of the API. If a client requests version 1, but version 2 is available, than the response SHOULD include a WernerLatestVersion: {value} in the response. The prefix of WERNER allows us to prevent any naming collisions if HTTP RFC changes occur.
To differentiate between different versions mapped to different methods and return types provide a headers parameter in the @RequestMapping
@RequestMapping(value = "/namespace/resource", method = RequestMethod.GET) public List<Resource> nameSpaceResource() throws Exception { return service.getResource(); }
@RequestMapping(value = "/namespace/resource", method = RequestMethod.GET, headers = {"Version=2"}) public List<Resource2> nameSpaceResourceV2() throws Exception { return service.getResource2(); }
The URI for each of the above endpoints is the same **/namespace/resource. ** nameSpaceResourceV2() though has a headers = {"version=2"}. If a client provides a header with a name of version and a value of 2 then the nameSpaceResourceV2() method will be executed and List<Resource2> will be returned. However, if no version header is provided OR a version header is provided with a value other than 2 - nameSpaceResource() will be executed.
Each endpoint will either be unversioned (Version 1) OR will have a version number. DO NOT version the whole API. There was some discussion around saying that a collection of Namespace endpoints should all be marketed as a general version of the full API. This has some drawbacks I will try and outline here.
- If over a period of time, we have had 3 version breaking changes to 3 endpoints. Say /ep1, /ep2 and ep3. /ep0 did not change.
- In essence, we would have gone from version 1 of the API to version 4.
- Each of the endpoint changes would have incremented the API version. So the API is at version 4.
- Client A requested the change to /ep3. They developed the ability to parse and handle the response of /ep3 (new version).
- Prior to that point, Client A was calling version 1 of the API because that was all they needed.
- Since Client A is now always supplying a v4 of the API and makes a request to /ep1 we need to decide which version of /ep1 to return.
- Since v4 is supplied in the header, we provide the largest match the changed version of /ep1.
- Client A is now broken unless they either -
- Change all their clients to upgrade all upgraded endpoints - adding a surprise scope to their development for something they did not need at the time
- For the /ep1 and /ep2 endpoints, client supplies a version appropriate header.
There is also implementation challenges with versioning the whole API.
- When /ep1 is first changed - it shifts to being handled as version 2 of /ep1.
- When /ep2 is first changed - it goes from version 1 to version 3
- When /ep3 is changed - it goes from version 1 to version 4
- There is a disconnect in the version numbering of the endpoints.
- When handling version numbers using Spring, a simple approach is to put the version number for a particular version in the headers of the @RequestMapping.
- If the full API version were changing - All endpoints with a version number would have to have their code touched to handle the new version of the API.
So with all this - we need to version each endpoint independently if it needs to change and break backward compatibility. Each endpoint supports a set of versions.
- Once an endpoint introduces it's third active version, a 6 month deadline must be provided to all clients to be calling the current version or current -1 version of the endpoint from a support standpoint.
- All endpoints less than current version -1 MUST BE marked as DEPRECATED.
All API requests need to be authenticated.
Authentication credentials MUST be provided in the Authentication header of the request. There are 2 potential options to use. Basic(User/Password) and Bearer(OAUth)
Use Basic when you have a small finite set of users that grow slowly and can be configured manually. Use Bearer with an OAuth2 implementation when the set of users grows dynamically, is likely to be a large set and is difficult to manage and maintain manually.
- A REST implementation SHOULD HAVE a response to Error! Hyperlink reference not valid. Swagger documentation with a list of all Endpoints, Models and descriptions within the domain.
A JSON object MUST be at the root of every JSON API request and response containing data. This object defines a document's "top level".
Implementing this guideline across products will standardize in house REST development. The guiding principal behind this guideline comes from HATEOS (Hypermedia as the Engine of Application State). A hypermedia-driven site provides information to navigate the site's REST interfaces dynamically by including hypermedia links with the responses.