Autobahn|Testsuite

The Autobahn|Testsuite provides a fully automated test suite to verify client and server implementations of The WebSocket Protocol for specification conformance and implementation robustness.

Autobahn|Testsuite also provides a couple of other tools useful for WebSocket (and WAMP) implementors and developers.

What users are saying:

"Autobahn was a huge help during the standardization and early development of WebSocket in Firefox. It helped find many nasty corner conditions in a clean and repeatable way. Interoperation testing like this really moves the Open Web forward." Patrick McManus, Mozilla Firefox

"The Jetty project has been active in the development of the websocket protocol and lack of a comprehensive test suite was an impediment to the standardisation of the protocol. The Autobahn test suit has filled that void and been warmly embraced by our websocket engineers at Intalio as it has been invaluable for evaluating the protocol development, improving inter operability and monitoring adoption of new/optional features. Our own implementation and the wider websocket 'ecosystem' are significantly better due to the availability of Autobahn." Greg Wilkins, Jetty author, Chief Architect at Intalio

"During the time of implementing the latest WebSocket version in Netty and Undertow we were searching for a way to test our implementations and make sure they are 100% RFC compliant. While we already had unit tests in place there was a lack of good tests for the whole implementation. Using the AutobahnTestsuite for this purpose allowed us to concentrate on the implementation and made it easy to catch regressions as soon as possible." Norman Maurer, Principal Software Engineer at Red Hat

Reports

For some current reports on the test coverage of Autobahn|Python see

Test Suite Coverage

The test suite will check an implementation by doing basic WebSocket conversations, extensive protocol compliance verification and performance and limits testing.

Autobahn|Testsuite is used across the industry and contains over 500 test cases covering

Framing
Pings/Pongs
Reserved Bits
Opcodes
Fragmentation
UTF-8 Handling
Limits/Performance
Closing Handshake
Opening Handshake (under development)
WebSocket compression (permessage-deflate extension)

Other Tools

Besides the automated testsuite (aka "fuzzing" server/client), wstest also includes a number of other handy modes:

WebSocket echo server and client
WebSocket broadcast server (and client driver)
Testee modes to test AutobahnPython against the test suite
wsperf controller and master (see below for more)
WAMP server and client, for developing WAMP implementations
WebSocket Mass-Connect

Users

Autobahn|Testsuite is used by numerous projects and companies across the industry:

Installation

Using the testsuite Docker image

The testsuite is available as a Docker image which allows easy and repeatable use both for testing WebSocket clients and WebSocket servers.

By default, the image will run the testsuite in so-called "fuzzingserver" mode. This is the mode used to test your WebSocket client implementations.

To start the testsuite container:

docker run -it --rm \
    -v "${PWD}/config:/config" \
    -v "${PWD}/reports:/reports" \
    -p 9001:9001 \
    --name fuzzingserver \
    crossbario/autobahn-testsuite

Above will mount an (included) test configuration from the docker/config folder, which must include a server test configuration file docker/config/fuzzingserver.json like this:

{
    "url": "ws://127.0.0.1:9001",
    "outdir": "./reports/clients",
    "cases": ["*"],
    "exclude-cases": [
        "9.*",
        "12.*",
        "13.*"
    ],
    "exclude-agent-cases": {}
}

This specific config will run all test cases, but exclude the longer running mass/performance test cases 9., and exclude the WebSocket compression test cases 12./13.* (which only make sense if your client library implements RFC7692 ("permessage-deflate")).

Above command will also mount a host directory/volume reports where the generated reports will be placed by the testsuite.

Finally, the config will make the fuzzing server run on port 9001 - and expose that on the host.

To test multiple clients and generate one big report containing all clients, do NOT stop the testsuite container, but run all your clients, and then stop the container. The generated reports will include all clients.

Here is how to test the Python 3 / asyncio flavor of AutobahnPython:

pip install autobahn
wget https://raw.githubusercontent.com/crossbario/autobahn-python/master/wstest/testee_client_aio.py
python testee_client_aio.py

The following recipe still works, but the new, recommended way is using a Docker toolchain image we provide. Please checkout this - much easier and repeatable.

The testsuite comes as a single command line tool, wstest. You will need Python 2 or PyPy (recommended).

Right now we only support Python 2 and Python 3 will not work. The testsuite is developed and tested on CPython 2 and PyPy. The latter is a high-performance Python implementation.

The recommended way to install wstest is into it's own, dedicated virtualenv.

On Debian/Ubuntu systems, you can install virtualenv like sudo apt-get install python-virtualenv.

Create a new virtualenv in your HOME and install Autobahn testsuite:

virtualenv ~/wstest
source ~/wstest/bin/activate
pip install autobahntestsuite

You will now have the wstest tool:

(wstest)oberstet@thinkpad-t430s:~$ which wstest
/home/oberstet/wstest/bin/wstest
(wstest)oberstet@thinkpad-t430s:~$ wstest -a
Autobahn 0.10.9
AutobahnTestSuite 0.7.4
(wstest)oberstet@thinkpad-t430s:~$ wstest --help
Usage: wstest [options]
Options:
  -d, --debug            Debug output [default: off].
  -a, --autobahnversion  Print version information for Autobahn and
                         AutobahnTestSuite.
  -m, --mode=            Test mode, one of: echoserver, echoclient,
                         broadcastclient, broadcastserver, fuzzingserver,
                         fuzzingclient, testeeserver, testeeclient, massconnect,
                         serializer [required]
  -t, --testset=         Run a test set from an import test spec.
  -s, --spec=            Test specification file [required in some modes].
  -o, --outfile=         Output filename for modes that generate testdata.
  -w, --wsuri=           WebSocket URI [required in some modes].
  -u, --webport=         Web port for running an embedded HTTP Web server;
                         defaults to 8080; set to 0 to disable. [optionally used
                         in some modes: fuzzingserver, echoserver,
                         broadcastserver, wsperfmaster]. [default: 8080]
  -i, --ident=           Testee client identifier [optional for client testees].
  -k, --key=             Server private key file for secure WebSocket (WSS)
                         [required in server modes for WSS].
  -c, --cert=            Server certificate file for secure WebSocket (WSS)
                         [required in server modes for WSS].
      --version          Display Twisted version and exit.
      --help             Display this help and exit.

(wstest)oberstet@thinkpad-t430s:~$

How to use

Testing WebSocket server implementations

To test a WebSocket server implementation and generate compliance test reports, first start the WebSocket server that you want to test. Here, we are using a example from Autobahn|Python:

(python279_1)oberstet@thinkpad-t430s:~/scm/crossbario/autobahn-python/examples/twisted/websocket/testee$ python testee_server.py
2015-12-21 21:31:54+0100 [-] Log opened.
2015-12-21 21:31:54+0100 [-] TesteeServerFactory starting on 9001
2015-12-21 21:31:54+0100 [-] Starting factory <__main__.TesteeServerFactory object at 0x7faf23551210>

Then, run wstest in fuzzing client mode:

cd ~
mkdir test
cd test
wstest -m fuzzingclient

The testsuite will now start a WebSocket fuzzing client connecting on TCP port 9001 to the WebSocket servers to be tested:

(wstest)oberstet@thinkpad-t430s:~/test$ wstest -m fuzzingclient
Auto-generating spec file 'fuzzingclient.json'
Loading spec from /home/oberstet/test/fuzzingclient.json
...
Autobahn Fuzzing WebSocket Client (Autobahn Version 0.7.4 / Autobahn Testsuite Version 0.10.9)
Ok, will run 521 test cases against 1 servers
...

Testing WebSocket client implementations

To test a WebSocket client implementation and generate compliance test reports, start wstest in fuzzing server mode:

cd ~
mkdir test
cd test
wstest -m fuzzingserver

The testsuite will now start a WebSocket fuzzing server listening on TCP port 9001 for WebSocket clients to be tested:

(wstest)oberstet@thinkpad-t430s:~/test$ wstest -m fuzzingserver
Auto-generating spec file 'fuzzingserver.json'
Loading spec from /home/oberstet/test/fuzzingserver.json
...
Autobahn WebSockets 0.7.4/0.10.9 Fuzzing Server (Port 9001)
Ok, will run 521 test cases for any clients connecting
...

Note: The fuzzing server mode will also start a Web server on port 8080 that renders a HTML page for browser WebSocket clients to be tested.

On first run, the tool will auto-generated a test configuration file:

(wstest)oberstet@thinkpad-t430s:~/test$ cat fuzzingserver.json

{
   "url": "ws://127.0.0.1:9001",
   "outdir": "./reports/clients",
   "cases": ["*"],
   "exclude-cases": [],
   "exclude-agent-cases": {}
}

You can tweak that file to run only some tests, e.g. "cases: ["1.*", "2.1.*"]" will run only the tests under section 1.* and subsection 2.1.*.

More Information

For more information take a look at the following information

usage instructions

Get in Touch

Get in touch on IRC #autobahn on chat.freenode.net or join an Autobahn user forum.

crossbario / autobahn-testsuite