vegardit / haxe-doctest

A haxelib inspired by Python's doctest command that generates unit tests based on assertions specified within the source code.

haxe-doctest - Haxedoc based unit testing.

1. What is it?
2. Declaring test assertions
3. Why doc-testing?
4. Doc-testing with Tink Testrunner
5. Doc-testing with Haxe Unit
6. Doc-testing with MUnit
7. Doc-testing with UTest
8. Doc-testing with hx.doctest.DocTestRunner
9. Doc-testing with FlashDevelop
10. Installation
11. Using the latest code
12. License

What is it?

A haxelib inspired by Python's doctest command that generates unit tests based on assertions declared within the Haxedoc comments of source code.

haxe-doctest supports the generation of test cases for Haxe Unit, MUnit, and it's own test runner which is recommended for efficient testing from within FlashDevelop.

Haxe compatiblity

haxe-doctest	Haxe
0.1.0 to 1.1.4	3.2.1 or higher
1.2.0 to 2.0.1	3.4.2 or higher
3.0.0 or higher	4.0.5 or higher

Declaring test assertions

Doc-test assertions are written as part of the source code documentation and are identified by three leading right angle brackets >>> before the assertion.

The left and the right side of the assertion must be separated by one of the comparison operators <, >, ==, !=, ===, !==, <=, >= or the throws keyword.

If the right side expression is a regeex, e.g. ~/my string/ then it will be matched against the string representation of the left side expression.

If a value to be checked needs to be calculated via multiple statements, they can be wrapped inside ({ }) where the last statement is the value to be checked. As an example, see the doctest for function MyObject.setData() below.

class MyTools {

   /**
    * <pre><code>
    * >>> MyTools.isValidName(null)   == false
    * >>> MyTools.isValidName("")     == false
    * >>> MyTools.isValidName("John") == true
    * </code></pre>
    */
   public static function isValidName(str:String):Bool {
      return str != null && str.length > 0;
   }
}


class MyObject {

   var data:String;

   /**
    * <pre><code>
    * >>> new MyObject(null) throws "[data] must not be null!"
    * >>> new MyObject(null) throws ~/must not be null/
    * >>> new MyObject("ab") throws nothing
    * </code></pre>
    */
   public function new(data:String) {
      if(data == null) throw "[data] must not be null!";
      this.data = data;
   }

   /**
    * <pre><code>
    * >>> new MyObject("ab").length()  > 1
    * >>> new MyObject("ab").length()  <= 2
    * >>> new MyObject("abc").length() != 2
    * </code></pre>
    */
   public function length():Int {
      return data == null ? 0 : data.length;
   }

   /**
    * <pre><code>
    * >>> ({
    * ...    var o=new MyObject("cat");
    * ...    o.setData("dog");
    * ...    o.data;  // return data property outside expression block for comparison
    * ... }) == "dog"
    * </code></pre>
    */
   public function setData(data:String):Void {
      this.data = data;
   }
}

Why doc-testing?

1. Doc-testing supports super fast test-driven development: First you write your method header, then the in-place documentation including your test assertions defining the expected behavior and then implement until all your declared tests pass.

   No need to create separate test classes with individual test methods. Implementing and testing happens at the same code location.

2. For users of your code, the doc-test assertions act as method documentation and code examples.

3. Since doc-testing actually means testing the documentation against the documented code, a method's documentation always represents the actual behavior of it's implementation and can't get accidently outdated.

Doc-testing with Haxe Unit

IMPORTANT: As of Haxe 4, Haxe Unit has been moved to a separate library called hx3compat. So you need to run haxe with -lib hx3compat

In the <project_root>/test folder, annotate a class extending haxe.unit.TestCase with @:build(hx.doctest.DocTestGenerator.generateDocTests()). The doc-test assertions from your source code will then be added as test methods to this class.

@:build(hx.doctest.DocTestGenerator.generateDocTests())
class MyHaxeUnitTest extends haxe.unit.TestCase {

   public static function main() {
      var runner = new haxe.unit.TestRunner();
      runner.add(new MyHaxeUnitTest());
      runner.run();
   }

   function new() {
      super();
   }
}

Doc-testing with Tink Testrunner

In the <project_root>/test folder, annotate a class extending tink.testrunner.BasicSuite with @:build(hx.doctest.DocTestGenerator.generateDocTests()). The doc-test assertions from your source code will then be added as test methods to this class.

@:build(hx.doctest.DocTestGenerator.generateDocTests())
class MyTinkTestrunnerTest extends tink.testrunner.Suite.BasicSuite {

   public static function main() {
      Runner.run(new MyTinkTestrunnerTest());
   }

   function new() {
      super({name: Type.getClassName(Type.getClass(this))}, []);
   }
}

Doc-testing with MUnit

In the <project_root>/test folder, annotate a test class with @:build(hx.doctest.DocTestGenerator.generateDocTests()). The doc-test assertions from your source code will then be added as test methods to this class.

@:build(hx.doctest.DocTestGenerator.generateDocTests())
class MyMUnitDocTests {
   public function new() { }
}

Then add the test class to a testsuite

class MyMUnitDocTestSuite extends massive.munit.TestSuite {
   public static function main() {
      var client = new massive.munit.RichPrintClient();
      var runner = new massive.munit.TestRunner(client);
      runner.run([MyMUnitDocTestSuite]);
   }

   public function new() {
      super();
      add(MyMUnitDocTests);
   }
}

Doc-testing with UTest

In the <project_root>/test folder, annotate a class extending utest.Test with @:build(hx.doctest.DocTestGenerator.generateDocTests()) AND @:build(utest.utils.TestBuilder.build()) - the order is important. The doc-test assertions from your source code will then be added as test methods to this class.

@:build(hx.doctest.DocTestGenerator.generateDocTests())
@:build(utest.utils.TestBuilder.build())
class MyUTestDocTests extends utest.Test {

   public static function main() {
      utest.UTest.run([new MyUTestDocTests()]);
   }

   function new() {
      super();
   }
}

Doc-testing with hx.doctest.DocTestRunner

haxe-doctest also comes with it's own Testrunner which is recommended for local testing as it generates console output that is parseable by FlashDevelop. When executed from within FlashDevelop, test failures will be displayed in the result panel as clickable errors that directly navigate your to the location in your source code.

In the <project_root>/test folder, annotate a class extending hx.doctest.DocTestRunner with @:build(hx.doctest.DocTestGenerator.generateDocTests()). The doc-test assertions from your source code will then be added as test methods to this class.

@:build(hx.doctest.DocTestGenerator.generateDocTests())
class MyDocTestRunner extends hx.doctest.DocTestRunner {

   public static function main() {
      var runner = new MyDocTestRunner();
      runner.runAndExit();
   }

   function new() { super(); }
}

To integrate this with FlashDevelop, create a batch file in your project root folder, e.g. called test-docs.cmd containing:

echo Compiling...
haxe -main mypackage.MyDocTestRunner ^
   -cp src ^
   -cp test ^
   -neko target/neko/MyDocTestRunner.n || goto :eof

echo Testing...
neko target/neko/TestRunner.n

In FlashDevelop create a new macro in the macro editor (which is reachable via the menu Macros -> Edit Macros...) containing the following statements.

InvokeMenuItem|FileMenu.Save
RunProcessCaptured|$(SystemDir)\cmd.exe;/c cd $(ProjectDir) & $(ProjectDir)\test-docs.cmd

Then assign the macro a short cut, e.g. [F4].

Now you can write your methods, document their behavior in the doc and by pressing [F4] your changes are saved and the doc-test assertions will be tested. Errors will show up as navigable events in the FlashDevelop's result panel.

Installation

1. install the library via haxelib using the command:

   haxelib install haxe-doctest
   

2. use in your Haxe project

   • for OpenFL/Lime projects add <haxelib name="haxe-doctest" /> to your project.xml
   • for free-style projects add -lib haxe-doctest to your *.hxml file or as command line option when running the Haxe compiler

Using the latest code

Using haxelib git

haxelib git haxe-doctest https://github.com/vegardit/haxe-doctest main D:\haxe-projects\haxe-doctest

Using Git

1. check-out the main branch

   git clone https://github.com/vegardit/haxe-doctest --branch main --single-branch D:\haxe-projects\haxe-doctest --depth=1
   

2. register the development release with haxe

   haxelib dev haxe-doctest D:\haxe-projects\haxe-doctest
   

License

All files are released under the Apache License 2.0.

Individual files contain the following tag instead of the full license text:

SPDX-License-Identifier: Apache-2.0

This enables machine processing of license information based on the SPDX License Identifiers that are available here: https://spdx.org/licenses/.