Go-FTW - Framework for Testing WAFs in Go!

This software should be compatible with the Python version.

I wrote this one to get more insights on the original version, and trying to shed some light on the internals. There are many assumptions on the inner workings that I needed to dig into the code to know how they worked.

My goals are:

get a compatible ftw version, with no dependencies and easy to deploy
be extremely CI/CD friendly
be fast (if possible)
add features like:
- syntax checking on the test files
- use docker API to get logs (if possible), so there is no need to read files
- add different outputs for CI (junit xml?, github, gitlab, etc.)

Install

Go to the releases page and get the one that matches your OS.

If you have Go installed and configured to run Go binaries from your shell you can also run

go install github.com/fzipi/go-ftw@latest

Example Usage

To run tests you need:

a WAF (doh!)
a file where the waf stores the logs
a config file, or environment variables, with the information to get the logs and how to parse them (I might embed this for the most commonly used, like Apache/NGiNX), and additional options described next.

YAML Config file

With a configuration file you can alter the test results, set paths for your environment, or enable features. The config file has four basic values:

logfile: <the relative path to the WAF logfile>
logmarkerheadername: <a header name used for log parsing (see "How log parsing works" below)>
testoverride: <a list of things to override (see "Overriding tests" below)>
mode: "default" or "cloud" (only change it if you need "cloud")

By default, ftw would search for a file in $PWD with the name .ftw.yaml. You can pass the --config <config file name> to point it to a different file.

Logfile

Running in default mode implies you have access to a logfile for checking the WAF behavior against test results. Example configurations for apache and nginx can be found below:

---
logfile: '../coreruleset/tests/logs/modsec2-apache/apache2/error.log'

---
logfile: '../coreruleset/tests/logs/modsec3-nginx/nginx/error.log'

WAF Server

I normally perform my testing using the Core Rule Set.

You can start the containers from that repo using docker compose:

git clone https://github.com/coreruleset/coreruleset.git
docker compose -f tests/docker-compose.yml up -d modsec2-apache

Running

This is the help for the run command:

❯ ftw run -h
Run all tests below a certain subdirectory. The command will search all y[a]ml files recursively and pass it to the test engine.

Usage:
  ftw run [flags]

Flags:
  -d, --dir string       recursively find yaml tests in this directory (default ".")
  -e, --exclude string   exclude tests matching this Go regexp (e.g. to exclude all tests beginning with "91", use "91.*").
                         If you want more permanent exclusion, check the 'testmodify' option in the config file.
  -h, --help             help for run
      --id string        (deprecated). Use --include matching your test only.
  -i, --include string   include only tests matching this Go regexp (e.g. to include only tests beginning with "91", use "91.*").
  -q, --quiet            do not show test by test, only results
  -t, --time             show time spent per test

Global Flags:
      --cloud           cloud mode: rely only on HTTP status codes for determining test success or failure (will not process any logs)
      --config string   override config file (default is $PWD/.ftw.yaml)
      --debug           debug output
      --trace           trace output: really, really verbose

Here's an example on how to run your tests:

ftw run -d tests -t

And the result should be similar to:

❯ ./ftw run -d tests -t

🛠️  Starting tests!
🚀 Running!
👉 executing tests in file 911100.yaml
	running 911100-1: ✔ passed 6.382692ms
	running 911100-2: ✔ passed 4.590739ms
	running 911100-3: ✔ passed 4.833236ms
	running 911100-4: ✔ passed 4.675082ms
	running 911100-5: ✔ passed 3.581742ms
	running 911100-6: ✔ passed 6.426949ms
...
	running 944300-322: ✔ passed 13.292549ms
	running 944300-323: ✔ passed 8.960695ms
	running 944300-324: ✔ passed 7.558008ms
	running 944300-325: ✔ passed 5.977716ms
	running 944300-326: ✔ passed 5.457394ms
	running 944300-327: ✔ passed 5.896309ms
	running 944300-328: ✔ passed 5.873305ms
	running 944300-329: ✔ passed 5.828122ms
➕ run 2354 total tests in 18.923445528s
⏭ skipped 7 tests
🎉 All tests successful!

Happy testing!

Additional features

templates with the power of Go text/template. Add your template to any data: sections and enjoy!
Sprig functions can be added to templates as well.
Override test results.
Cloud mode! This new mode will override test results and rely solely on HTTP status codes for determining success and failure of tests.

With templates and functions you can simplify bulk test writing, or even read values from the environment while executing. This features allow you to write tests like this:

data: 'foo=%3d{{ "+" | repeat 34 }}'

Will be expanded to:

data: 'foo=%3d++++++++++++++++++++++++++++++++++'

But also, you can get values from the environment dynamically when the test is run:

data: 'username={{ env "USERNAME" }}

Will give you, as you expect, the username running the tests

data: 'username=fzipi

Other interesting functions you can use are: randBytes, htpasswd, encryptAES, etc.

Overriding tests

Sometimes you have tests that work well for some platform combinations, e.g. Apache + modsecurity2, but fail for others, e.g. NGiNX + modsecurity3. Taking that into account, you can override test results using the testoverride config param. The test will be run, but the result would be overriden, and your comment will be printed out.

Tests can be altered using four lists:

input allows you to override global parameters in tests. An example usage is if you want to change the dest_addr of all tests to point to an external IP or host
ignore is for tests you want to ignore. It will still execute the test, but ignore the result. You should add a comment on why you ignore the test
forcepass is for tests you want to pass unconditionally. Test will be executed, and pass even when the test fails. You should add a comment on why you force pass the test
forcefail is for tests you want to fail unconditionally. Test will be executed, and fail even when the test passes. You should add a comment on why you force fail the test

Example using all the lists above:

...
testoverride:
  input:
    dest_addr: "192.168.1.100"
    port: 8080
    protocol: "http"
  ignore:
    # text comes from our friends at https://github.com/digitalwave/ftwrunner
    '941190-3': 'known MSC bug - PR #2023 (Cookie without value)'
    '941330-1': 'know MSC bug - #2148 (double escape)'
    '942480-2': 'known MSC bug - PR #2023 (Cookie without value)'
    '944100-11': 'known MSC bug - PR #2045, ISSUE #2146'
  forcefail:
    '123456-01': 'I want this test to fail, even if passing'
  forcepass:
    '123456-02': 'This test will always pass'

You can combine any of ignore, forcefail and forcepass to make it work for you.

☁️ Cloud mode

Most of the tests rely on having access to a logfile to check for success or failure. Sometimes that is not possible, for example, when testing cloud services or servers where you don't have access to logfiles and/or logfiles won't have the information you need to decide if the test was good or bad.

With cloud mode, we move the decision on test failure or success to the HTTP status code received after performing the test. The general idea is that you setup your WAF in blocking mode, so anything matching will return a block status (e.g. 403), and if not we expect a 2XX return code.

An example config file for this is:

---
mode: 'cloud'

Or you can just run: ./ftw run --cloud

How log parsing works

The log output from your WAF is parsed and compared to the expected output. The problem with log files is that they aren't updated in real time, e.g. because the web server / WAF has an internal buffer, or because there's some fsync magic involved). To make log parsing consistent and guarantee that we will see output when we need it, go-ftw uses "log markers". In essence, unique log entries are written before and after every test stage. go-ftw can then search for these markers.

The container images for Core Rule Set can be configured to write these marker log lines by setting the CRS_ENABLE_TEST_MARKER environment variable. If you are testing a different WAF you will need to instrument it with the same idea (unless you are using "cloud mode"). The rule for CRS looks like this:

# Write the value from the X-CRS-Test header as a marker to the log
SecRule REQUEST_HEADERS:X-CRS-Test "@rx ^.*$" \
  "id:999999,\
  phase:1,\
  log,\
  msg:'X-CRS-Test %{MATCHED_VAR}',\
  pass,\
  t:none"

The rule looks for an HTTP header named X-CRS-Test and writes its value to the log, the value being the UUID of a test stage.

You can configure the name of the HTTP header by setting the logmarkerheadername option in the configuration to a custom value (the value is case insensitive).

marcsvll / go-ftw