This repo includes a pytest-wdl skeleton for developing test cases for the WDL language specification. While this is suggested as a "shovel-ready" starting point for participants, other testing frameworks and approaches are equally welcomed, as one of the Testathon's goals is to develop general best practices for WDL testing.
Platform requirements:
| Linux | macOS | |
|---|---|---|
| Python 3.6+ & pip | e.g. apt-get install python3 python3-pip |
install from Homebrew (Python3.framework may work, with caveat) |
| docker | (i) e.g. apt-get install docker.io, (ii) empower user to control it |
set up Docker Desktop for Mac |
| Java 8+ | e.g. apt-get install default-jre-headless |
run java in Terminal to trigger dialog (alternative: AdoptOpenJDK) |
Booting up:
- (if desired) create & enter a python virtualenv
- Fork this GitHub repository
- Clone your fork to working directory:
git clone git@github.com:GITHUB_USERNAME/Testathon-2020.git && cd Testathon-2020 - Add upstream for later use:
git remote add upstream https://github.com/openwdl/Testathon-2020.git - Install pytest-wdl and any other dependencies:
pip3 install --user -r requirements.txt - Download Cromwell JAR to some known location
cp pytest_wdl_config.json $HOMEand edit the copy so thatcromwell_jar_filehas the correct location
pytest .
Once comfortable, run tests in parallel to reduce total time:
pytest -n 10 -v .
Or run just one test:
pytest tests/expr/optional_interpolation
We've seeded four broad categories of test cases in this repository:
tests/expr: parsing, type-checking, and evaluation of basic WDL expressionstests/stdlib: standard library functionstests/task: task-level runner functionalitytests/workflow: workflow-level runner functionality
These categories reflect the intended focus within, but aren't mutually exclusive (for example, a scatter{} workflow can be useful to iterate more-basic tests on varying inputs). More categories might be needed, such as for structs and imports.
Each folder within these categories provides a pytest-wdl test case including,
- .wdl file
- short Python file describing how to invoke the workflow and its expected outputs (with the
workflow_runnerfixture) - optionally, a
test-data.jsonfile describing any needed data assets (with theworkflow_datafixture)
The pytest-wdl manual has full detail on these fixtures.
New tests can be added by copying an existing test case directory alongside, and renaming+rewriting the files within as needed. Pytest automatically discovers all test_*.py files in the directory tree.
Documentation: at minimum, each Python file should include a docstring describing the focus.
Send pull requests with new test cases upstream via GitHub. And sync up with the latest from time to time, git fetch upstream && git merge upstream/master (rebase if you dare).