Add "only" mode

[Francisco Ryan Tolmasky I]

We need a way to specify that you should only be doing one of the tests in the file for quick testing. Perhaps something like [only] in the title, and then require --allow-only as a flag. This way, it won't get accidentally committed.

[Noah Tye]

I think the best solution is for an --only parameter that accepts a regex, and only tests whose titles match the regex get run.

This way, we know nothing gets committed, plus the decision of whether or not to run a given test relies only on that test and the regex (ie, you don't have to know anything about any other tests to make the decision).

This is what the Go testing command does, and I think it works pretty well.

[Francisco Ryan Tolmasky I]

We have to at minimum calculate the path to the test and run all the preceding requirements (previous tests in serial Suites, and parent Suites).

[Noah Tye]

Ah, right, test dependencies (parents, preceding serial tests) do need to be determined and run.

Why do we need to calculate the path to the test?

[Francisco Ryan Tolmasky I]

So that we don't run unecessary tests. For example, if you have Suite A { Suite B { Test 1, Test 2 } }, merely knowing Suite A is the parent to start with isn't sufficient, since you need to know that you SHOULD run Suite B but NOT Test 2, so you'd want, working backwards from the matched regex, A -> B -> 1. If it's Suite A { Suite B { ... } . Suite C (Serial) { Test 1, Test 2 } }, you want A -> C -> 1 -> 2. You then want to merge the paths of all the matches, so that you have have a green-lit set to compare to when walking the test tree as normal.

[Noah Tye]

Oh oh, sorry, I misinterpreted that to mean "the path on the filesystem to the file which contains the test". Got it, yes that makes sense.

[Francisco Ryan Tolmasky I]

This is just conceptually of course, you can imagine just doing getPrunedTree:

getPrunedTree (node)
{
     if (node.title matches regex)
         return node;

     if (node is Suite)
     {
          children = node.children.filter(getPrunedTree);
          if (children.length > 0)
              return Suite({ ...node, children });
         else
              return null;
     }
}