Copyright 2017-2021 DMTF. All rights reserved.
The Redfish Service Validator
is a python3 tool for checking conformance of any "device" with a Redfish service interface against Redfish CSDL schema. The tool is designed to be device agnostic and is driven based on the Redfish specifications and schema intended to be supported by the device.
The Redfish Service Validator
is an open source framework for checking conformance of any generic device with Redfish interface enabled against the DMTF defined Redfish schema and specifications. The tool is designed to be device agnostic and is driven purely based on the Redfish specifications intended to be supported by the device.
The Redfish Service Validator
is based on Python 3 and the client system is required to have the Python framework installed before the tool can be installed and executed on the system. Additionally, the following packages are required to be installed and accessible from the python environment:
- beautifulsoup4 - https://pypi.python.org/pypi/beautifulsoup4
- requests - https://github.com/kennethreitz/requests (Documentation is available at http://docs.python-requests.org/)
- lxml - https://pypi.python.org/pypi/lxml
You may install the prerequisites by running:
pip3 install -r requirements.txt
If you have a previous beautifulsoup4 installation, please use the following command:
pip3 install beautifulsoup4 --upgrade
There is no dependency based on Windows or Linux OS. The result logs are generated in HTML format and an appropriate browser (Chrome, Firefox, IE, etc.) is required to view the logs on the client system.
Place the RedfishServiceValidator folder into the desired directory. Create the following subdirectories in the tool root directory: "config", "logs", "SchemaFiles". Place the example config.ini file in the "config" directory. Place the CSDL Schema files to be used by the tool in the root of the schema directory, or the directory given in config.ini.
The Redfish Service Validator
is designed to execute as a purely command line interface tool with no intermediate inputs expected during tool execution. However, the tool requires various inputs regarding system details, DMTF schema files etc. which are consumed by the tool during execution to generate the conformance report logs. Below are the step by step instructions on setting up the tool for execution on any identified Redfish device for conformance test:
Modify the config\example.ini file to enter the system details under below section
Variable | Type | Definition |
---|---|---|
Version | string | Internal config version (optional) |
Copyright | string | DMTF copyright (optional) |
verbose | int | level of verbosity (0-3) |
Variable | Type | Definition |
---|---|---|
ip | string | Host of testing system, formatted as https:// ip : port (can use http as well) |
username | string | Username for Basic authentication |
password | string | Password for Basic authentication (removed from logs) |
description | string | Description of system being tested (optional) |
forceauth | boolean | Force authentication even on http servers |
authtype | string | Authorization type (Basic |
token | string | Token string for Token authentication |
Variable | Type | Definition |
---|---|---|
payload | string | Option to test a specific payload or resource tree (see below) |
logdir | string | Place to save logs and run configs |
oemcheck | boolean | Whether to check Oem items on service |
debugging | boolean | Whether to print debug to log |
schema_directory | string | Where schema is located/saved on system |
The payload option takes two parameters as "option uri"
(Single, SingleFile, Tree, TreeFile) How to test the payload URI given. Single tests will only give a report on a single resource, while Tree will report on every link from that resource
([Filename], [uri])
URI of the target payload, or filename of a local file.
Redfish Service Validator
starts with the Service root Resource Schema by querying the service with the service root URI and getting all the device information, the resources supported and their links. Once the response of the Service root query is verified against its schema, the tool traverses through all the collections and Navigation properties returned by the service.- From the Metadata, collect all XML specified possible from the service, and store them in a tool-specified directory
- For each navigation property/Collection of resource returned, it does following operations:
- Reads all the Navigation/collection of resources from the respective resource collection schema file.
- Reads the schema file related to the particular resource, collects all the information about individual properties from the resource schema file and stores them into a dictionary
- Queries the service with the individual resource uri and validates all the properties returned against the properties collected from the schema file using a GET method making sure all the Mandatory properties are supported
- Step 2 repeats until all of the URIs and resources are covered.
Upon validation of a resource, the following types of tests may occur:
- Upon reaching any resource, validate its @odata entries inside of its payload with regex. If it fails, return a "failPayloadError".
- Attempt to initiate a Resource object, which requires an @odata.type and Schema of a valid JSON payload, otherwise return a "problemResource" or "exceptionResource" and terminate, otherwise "passGet"
- With the Resource initiated, begin to validate each Property available based on its Schema definition (sans annotations, additional props, is Case-Sensitive):
- Check whether a Property is at first able to be nulled or is mandatory, and pass based on its Requirement or Nullability
- For collections, validate each property inside of itself, and expects a list rather than a single Property, otherwise validate normally:
- For basic types such as Int, Bool, DateTime, GUID, String, etc, check appropriate types, regex and ranges.
- For Complex types, validate each property inside of the Complex, including annotations and additional properties
- For Enum types, check and see if the given value exists in Schema's defined EnumType (Case-Sensitive)
- For Entity types, check if the link given by @odata.id sends the client to an appropriate resource by the correct type, by performing a GET
- After reaching completion, perform the same routine with all Annotations available in the payload.
- If any unvalidated entries exist in the payload, determine whether or not additional properties are legitimate for this resource, otherwise throw a "failAdditional" error.
The Redfish Service Validator
generates an html report under the “logs” folder, named as ConformanceHtmlLog_MM_DD_YYYY_HHMMSS.html, along with a text and config file. The report gives the detailed view of the individual properties checked, with the Pass/Fail/Skip/Warning status for each resource checked for conformance.
Additionally, there is a verbose log file that may be referenced to diagnose tool or schema problems when the HTML log is insufficient.
The test result for each GET operation will be reported as follows:
- PASS: If the operation is successful and returns a success code (E.g. 200, 204)
- FAIL: If the operation failed for reasons mentioned in GET method execution, or some configuration.
- SKIP: If the property or method being checked is not mandatory is not supported by the service.
Redfish Service Validator
covers all the GET execution on the service. Below are certain points which are not in this scope.
- Patch/Post/Skip/Top/Head is not covered as part of
Redfish Service Validator
due to dependency on internal factor of the service. Redfish Service Validator
does not cover testing of multiple service at once. To execute this, we have to re-run the tool by running it separately.- Tool doesn't support @odata.context which use complex $entity path
The module pyinstaller is used to package the environment as a standlone executable file; this can be installed with the following command:
pip3 install pyinstaller
From a Windows system, the following command can be used to build a Windows executable file named RedfishServiceValidator.exe
, which will be found in dist folder:
pyinstaller -F -w -i redfish.ico -n RedfishServiceValidator.exe RedfishServiceValidatorGui.py
- Update
CHANGELOG.md
with the list of changes since the last release - Update the
tool_version
variable inRedfishServiceValidator.py
to reflect the new tool version - Push changes to Github
- Create the standalone executable as described in the above section
- Create the zip file for the release containing the generated exe file and
README.md
- Name format: Redfish-Service-Validator-X.Y.Z-Windows.zip
- Create a new release in Github with the zip file attached