Fluster

Fluster is a testing framework written in Python for decoders conformance. It is composed of a CLI application that runs a number of test suites with the supported decoders. Its purpose is to check different decoder implementations against known test suites with known and tested results. It has been originally designed to check the conformance of H.265/HEVC decoders, but it also supports H.264/AVC and can be easily extended to add more decoders and test suites.

Fluster

Description

Fluster requires Python 3.6+ to work. It has zero dependencies apart from that. The requirements.txt file includes Python's modules used only for development.

The framework works with test suites. Each test suite is associated with one codec and contains a number of test vectors. Each test vector consists of an input file and and the expected result. The input file will be fed into each decoder that supports the codec of the test suite. The file format is a JSON file. You can find the ones included in the test_suites directory.

The decoders are the ones in charge of doing the decoding given an input file. They implement two methods: decode which is mandatory and check which is optional. Check out the decoder class for reference. The @register_decoder decorator is used to ensure the framework takes them into account. Fluster is agnostic as to how the decoding itself is done. So far, all decoders are external processes that need to run with a number of parameters, but they could actually be decoders written in Python as far as Fluster is concerned. The decoders directory contains all supported decoders.

In order to run the tests for the different test suites and decoders, a resources directory containing all the input files for each test suite needs to be collected first.

How to use

Download the resources for the test suites. This will create the resources directory containing the input files for each test suite. ./fluster.py download will spawn a number of processes to download and extract the files. You can change the number of parallel processes used with -j. It defaults to 2x number of logical cores.
(Optional) Build the reference decoders for H.264/AVC and H.265/HEVC running make decoders. It assumes you have CMake and a native compiler such as gcc or clang installed so that they can be built. The resulting binaries will be moved to a new decoders directory in the root.
List the test suites, the decoders and which of them can run using ./fluster.py list -c.

List of available test suites:

VP8-TEST-VECTORS
    Codec: VP8
    Description: VP8 Test Vector Catalogue from https://github.com/webmproject/vp8-test-vectors
    Test vectors: 61

JVT-AVC_V1
    Codec: H.264
    Description: JVT AVC version 1
    Test vectors: 135

JCT-VC-HEVC_V1
    Codec: H.265
    Description: JCT-VC HEVC version 1
    Test vectors: 147

VP9-TEST-VECTORS
    Codec: VP9
    Description: VP9 Test Vector Catalogue from https://storage.googleapis.com/downloads.webmproject.org/test_data/libvpx/
    Test vectors: 303

List of available decoders:

DUMMY
    Dummy: This is a dummy implementation for the dummy codec... ✔️

H264
    FFmpeg-H.264: FFmpeg H.264 SW decoder... ✔️
    FFmpeg-H.264-D3D11VA: FFmpeg H.264 D3D11VA decoder... ❌
    FFmpeg-H.264-DXVA2: FFmpeg H.264 DXVA2 decoder... ❌
    FFmpeg-H.264-VAAPI: FFmpeg H.264 VAAPI decoder... ✔️
    FFmpeg-H.264-VDPAU: FFmpeg H.264 VDPAU decoder... ✔️
    Fluendo-H.264-HW-Gst1.0: Fluendo H.264 HW decoder for GStreamer 1.0... ❌
    Fluendo-H.264-HW-vah264dec-Gst1.0: Fluendo H.264 HW decoder for GStreamer 1.0... ❌
    Fluendo-H.264-SW-Gst0.10: Fluendo H.264 SW decoder for GStreamer 0.10... ❌
    Fluendo-H.264-SW-Gst1.0: Fluendo H.264 SW decoder for GStreamer 1.0... ❌
    GStreamer-H.264-D3D11-Gst1.0: GStreamer H.264 D3D11 decoder for GStreamer 1.0... ❌
    GStreamer-H.264-Libav-Gst1.0: GStreamer H.264 Libav decoder for GStreamer 1.0... ✔️
    GStreamer-H.264-MSDK-Gst1.0: GStreamer H.264 MSDK decoder for GStreamer 1.0... ❌
    GStreamer-H.264-NVDEC-Gst1.0: GStreamer H.264 NVDEC decoder for GStreamer 1.0... ❌
    GStreamer-H.264-V4L2SL-Gst1.0: GStreamer H.264 V4L2SL decoder for GStreamer 1.0... ❌
    GStreamer-H.264-VA-Gst1.0: GStreamer H.264 VA decoder for GStreamer 1.0... ❌
    GStreamer-H.264-VAAPI-Gst1.0: GStreamer H.264 VAAPI decoder for GStreamer 1.0... ✔️
    JCT-VT-H.264: JCT-VT H.264/AVC reference decoder... ✔️

H265
    FFmpeg-H.265: FFmpeg H.265 SW decoder... ✔️
    FFmpeg-H.265-D3D11VA: FFmpeg H.265 D3D11VA decoder... ❌
    FFmpeg-H.265-DXVA2: FFmpeg H.265 DXVA2 decoder... ❌
    FFmpeg-H.265-VAAPI: FFmpeg H.265 VAAPI decoder... ✔️
    FFmpeg-H.265-VDPAU: FFmpeg H.265 VDPAU decoder... ✔️
    Fluendo-H.265-HW-vah265dec-Gst1.0: Fluendo H.265 HW decoder for GStreamer 1.0... ❌
    Fluendo-H.265-SW-Gst0.10: Fluendo H.265 SW decoder for GStreamer 0.10... ❌
    Fluendo-H.265-SW-Gst1.0: Fluendo H.265 SW decoder for GStreamer 1.0... ❌
    Fluendo-H.265-byte-stream-au-HW-Gst1.0: Fluendo H.265-byte-stream-au HW decoder for GStreamer 1.0... ❌
    Fluendo-H.265-byte-stream-nal-HW-Gst1.0: Fluendo H.265-byte-stream-nal HW decoder for GStreamer 1.0... ❌
    Fluendo-H.265-hev1-au-HW-Gst1.0: Fluendo H.265-hev1-au HW decoder for GStreamer 1.0... ❌
    Fluendo-H.265-hev1-nal-HW-Gst1.0: Fluendo H.265-hev1-nal HW decoder for GStreamer 1.0... ❌
    Fluendo-H.265-hvc1-au-HW-Gst1.0: Fluendo H.265-hvc1-au HW decoder for GStreamer 1.0... ❌
    Fluendo-H.265-hvc1-nal-HW-Gst1.0: Fluendo H.265-hvc1-nal HW decoder for GStreamer 1.0... ❌
    GStreamer-H.265-D3D11-Gst1.0: GStreamer H.265 D3D11 decoder for GStreamer 1.0... ❌
    GStreamer-H.265-Libav-Gst1.0: GStreamer H.265 Libav decoder for GStreamer 1.0... ✔️
    GStreamer-H.265-MSDK-Gst1.0: GStreamer H.265 MSDK decoder for GStreamer 1.0... ❌
    GStreamer-H.265-NVDEC-Gst1.0: GStreamer H.265 NVDEC decoder for GStreamer 1.0... ❌
    GStreamer-H.265-V4L2SL-Gst1.0: GStreamer H.265 V4L2SL decoder for GStreamer 1.0... ❌
    GStreamer-H.265-VA-Gst1.0: GStreamer H.265 VA decoder for GStreamer 1.0... ❌
    GStreamer-H.265-VAAPI-Gst1.0: GStreamer H.265 VAAPI decoder for GStreamer 1.0... ✔️
    JCT-VT-H.265: JCT-VT H.265/HEVC reference decoder... ✔️

VP8
    FFmpeg-VP8: FFmpeg VP8 SW decoder... ✔️
    FFmpeg-VP8-VAAPI: FFmpeg VP8 VAAPI decoder... ✔️
    GStreamer-VP8-D3D11-Gst1.0: GStreamer VP8 D3D11 decoder for GStreamer 1.0... ❌
    GStreamer-VP8-Libav-Gst1.0: GStreamer VP8 Libav decoder for GStreamer 1.0... ✔️
    GStreamer-VP8-V4L2SL-Gst1.0: GStreamer VP8 V4L2SL decoder for GStreamer 1.0... ❌
    GStreamer-VP8-VA-Gst1.0: GStreamer VP8 VA decoder for GStreamer 1.0... ❌
    GStreamer-VP8-VAAPI-Gst1.0: GStreamer VP8 VAAPI decoder for GStreamer 1.0... ✔️
    GStreamer-VP8-libvpx-Gst1.0: GStreamer VP8 libvpx decoder for GStreamer 1.0... ✔️
    libvpx-VP8: VP8 reference decoder... ✔️

VP9
    FFmpeg-VP9: FFmpeg VP9 SW decoder... ✔️
    FFmpeg-VP9-VAAPI: FFmpeg VP9 VAAPI decoder... ✔️
    GStreamer-VP9-D3D11-Gst1.0: GStreamer VP9 D3D11 decoder for GStreamer 1.0... ❌
    GStreamer-VP9-Libav-Gst1.0: GStreamer VP9 Libav decoder for GStreamer 1.0... ✔️
    GStreamer-VP9-V4L2SL-Gst1.0: GStreamer VP9 V4L2SL decoder for GStreamer 1.0... ❌
    GStreamer-VP9-VA-Gst1.0: GStreamer VP9 VA decoder for GStreamer 1.0... ❌
    GStreamer-VP9-VAAPI-Gst1.0: GStreamer VP9 VAAPI decoder for GStreamer 1.0... ✔️
    GStreamer-VP9-libvpx-Gst1.0: GStreamer VP9 libvpx decoder for GStreamer 1.0... ✔️
    libvpx-VP9: VP9 reference decoder... ✔️

Run the test suite (or a number of them) for all decoders (or a number of them). By default, decoder test are run in parallel. By default it uses the same amount of parallel jobs as number of cores, but it can be configured using the -j option. You can pass -d to filter only the decoders that you want to run, -ts for the test suites and -tv for the test vectors. Examples:
- ./fluster.py run runs all test suites for all decoders available that match each test suite's codec.
- ./fluster.py run -ts JCT-VC-HEVC_V1 runs the JCT-VC-HEVC_V1 test suite for all decoders that support H.265/HEVC.
- ./fluster.py run -ts JCT-VC-HEVC_V1 -tv AMP_A_Samsung_7 runs only the test vector AMP_A_Samsung_7 of the JCT-VC-HEVC_V1 test suite.
- ./fluster.py run -d FFmpeg-H265 runs the FFmpeg-H265 decoder on all test suites for H.265/HEVC.
- ./fluster.py run -d FFmpeg-H265 -j1 runs the FFmpeg-H265 decoder on all test suites for H.265/HEVC using one job.

Test Suites

Dummy test suite for testing purposes.
H.264/AVC.
H.265/HEVC.
VP8.
VP9.

Decoders

Dummy decoder for testing purposes.
JCT-VT H.265/HEVC as reference decoder for H.265/HEVC.
JCT-VT H.264/AVC as reference decoder for H.264/AVC.
Fluendo's propietary decoders for H.264/AVC and H.265/HEVC that are included in Fluendo Codec Pack.
GStreamer's for H.265/HEVC.
GStreamer's for H.264/AVC.
FFmpeg's for H.265/HEVC.
FFmpeg's for H.264/AVC.
libvpx's for VP8.
libvpx's for VP9.

CLI usage

./fluster.py --help

usage: fluster.py [-h] [-r RESOURCES] [-o OUTPUT] [-ne] [-tsd TEST_SUITES_DIR] {list,l,run,r,download,d,reference} ...

optional arguments:
  -h, --help            show this help message and exit
  -r RESOURCES, --resources RESOURCES
                        set the directory where resources are taken from
  -o OUTPUT, --output OUTPUT
                        set the directory where test results will be stored
  -ne, --no-emoji       set to use plain text instead of emojis
  -tsd TEST_SUITES_DIR, --test-suites-dir TEST_SUITES_DIR
                        set the directory where test suite will be read

subcommands:
  {list,l,run,r,download,d,reference}
    list (l)            show list of available test suites and decoders
    run (r)             run test suites for decoders
    download (d)        downloads test suites resources
    reference (r)       use a specific decoder to set its results for the test suites given

List

./fluster.py list --help

usage: fluster.py list [-h] [-ts TESTSUITES [TESTSUITES ...]] [-tv] [-c] [-v]

optional arguments:
  -h, --help            show this help message and exit
  -ts TESTSUITES [TESTSUITES ...], --testsuites TESTSUITES [TESTSUITES ...]
                        show only the test suites given
  -tv, --testvectors    show test vectors of test suites
  -c, --check           check which decoders can be run successfully. Reports ✔️ or ❌
  -v, --verbose         show stdout and stderr of commands executed

Run

./fluster.py run --help

usage: fluster.py run [-h] [-j JOBS] [-t TIMEOUT] [-ff] [-q] [-ts TESTSUITES [TESTSUITES ...]] [-tv TESTVECTORS [TESTVECTORS ...]]
                      [-d DECODERS [DECODERS ...]] [-s] [-so SUMMARY_OUTPUT] [-k] [-th THRESHOLD] [-tth TIME_THRESHOLD] [-v]

optional arguments:
  -h, --help            show this help message and exit
  -j JOBS, --jobs JOBS  number of parallel jobs to use. 1x logical cores by default.0 means all logical cores
  -t TIMEOUT, --timeout TIMEOUT
                        timeout in secs for each decoding. Defaults to 30 secs
  -ff, --failfast       stop after first fail
  -q, --quiet           don't show every test run
  -ts TESTSUITES [TESTSUITES ...], --testsuites TESTSUITES [TESTSUITES ...]
                        run only the specific test suites
  -tv TESTVECTORS [TESTVECTORS ...], --testvectors TESTVECTORS [TESTVECTORS ...]
                        run only the specific test vectors
  -d DECODERS [DECODERS ...], --decoders DECODERS [DECODERS ...]
                        run only the specific decoders
  -s, --summary         generate a summary in Markdown format for each test suite
  -so SUMMARY_OUTPUT, --summary-output SUMMARY_OUTPUT
                        dump summary output to file
  -f {md,csv,junitxml}, --format {md,csv,junitxml}
                        specify the format for the summary output. Markdown is selected by default
  -k, --keep            keep output files generated during the test
  -th THRESHOLD, --threshold THRESHOLD
                        set exit code to 2 if threshold tests are not success. exit code is 0 otherwise
  -tth TIME_THRESHOLD, --time-threshold TIME_THRESHOLD
                        set exit code to 3 if test suite takes longer than treshold seconds. exit code is 0 otherwise
  -v, --verbose         show stdout and stderr of commands executed

Download

./fluster.py download --help

usage: fluster.py download [-h] [-j JOBS] [-k] [testsuites ...]

positional arguments:
  testsuites            list of testsuites to download

optional arguments:
  -h, --help            show this help message and exit
  -j JOBS, --jobs JOBS  number of parallel jobs to use. 2x logical cores by default.0 means all logical cores
  -k, --keep            keep downloaded file after extracting

Reference

./fluster.py reference --help

usage: fluster.py reference [-h] [-j JOBS] [-t TIMEOUT] [-q] [-v] decoder testsuites [testsuites ...]

positional arguments:
  decoder               decoder to run
  testsuites            list of testsuites to run the decoder with

optional arguments:
  -h, --help            show this help message and exit
  -j JOBS, --jobs JOBS  number of parallel jobs to use. 1x logical cores by default.0 means all logical cores
  -t TIMEOUT, --timeout TIMEOUT
                        timeout in secs for each decoding. Defaults to 30 secs
  -q, --quiet           don't show every test run
  -v, --verbose         show stdout and stderr of commands executed

Report

Go to report

FAQ

Where does the name come from?

Fluster in English means to (cause to) become nervous or confused. It looks a very appropriate name for testing decoders.

How can I add a new decoder?

Create a new decoder in fluster/decoders directory.
Implement the decode method.
Use the register_decoder decorator.
Ensure to set hw_acceleration = True if it requires hardware.
Optionally, implement the check to know whether the decoder is available to be run.

How can I create a new test suite?

Check out the JSON format they follow in the test_suites directory. Add a new json file within and Fluster will automatically pick it up.

There is also a generator script for the conformance test suites that you can use as a base to generate automatically new ones.

How can I use it to test regressions?

We can easily use Fluster in a CI to test that our test suites for particular decoders are still working as good as they were before. There are two arguments to be used with the run command that can help us achieve that. Both commands work only when running a single test suite:

-th/--threshold sets the minimum number of tests that need to be success in order to consider the command not a failure. In case of failure, the exit code is 2. Please notice that even of some tests fail, the exit code will still be 0 as long as the threshold is met.
-tth/--time-threshold sets the maximum amount of time for a test suite to run and be considered a success. The exit code 3 in case it takes longer and 0 otherwise. Please notice that even of some tests fail, the exit code will still be 0 as long as the time it takes is less than the threshold.

However, in case we want to be even more explicit, ensuring exactly the same results are obtained, you can do the following procedure:

Store well known-results in an expected file with the output of Fluster's run.
Run Fluster again and compare with the expected result using diff. The following command will return an exitStatus different from 0 in case there's any difference.
```
diffParams='-uB -I"|Test|" --strip-trailing-cr'
diff $diffParams expected.out current.out
```
We recommend using the --no-emoji argument when running on the CI to ensure the log is properly displayed with no issues.

How can I contribute?

Fork the repo.
Install the required Python modules for development using pip3 install -r requirements.txt.
Install the git hook that will run for every commit to ensure it works before pushing. About git hooks.
Modify the code. Make sure the git hook is properly checking that the basic functionality still works. You can also execute make check manually. Take into account that some basic tests are run to ensure that GStreamer, FFmpeg and libvpx decoders are working fine. Thus, you need to have them available. On an Ubuntu 20.04 the needed packages are:
```
gstreamer1.0-tools gstreamer1.0-libav gstreamer1.0-plugins-bad ffmpeg vpx-tools
```
Create a new PR with your changes.
Make sure the GitHub Actions is running and its result is a pass.

How can I report an issue?

In case you find any problem or want to report something, don't hesitate to search for similar issues. Only when the issue can't be found, a new one should be created. Please try to provide as many details and context as possible to help us diagnose it.

License

LGPLv3

Fluster - testing framework for decoders conformance
Copyright (C) 2020, Fluendo, S.A.
  Author: Pablo Marcos Oltra <pmarcos@fluendo.com>, Fluendo, S.A.
  Author: Andoni Morales Alastruey <amorales@fluendo.com>, Fluendo, S.A.

This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Library General Public
License as published by the Free Software Foundation; either
version 2 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
Library General Public License for more details.

You should have received a copy of the GNU Library General Public
License along with this library; if not, write to the
Free Software Foundation, Inc., 59 Temple Place - Suite 330,
Boston, MA 02111-1307, USA.

jernejsk / fluster