bitwes / Gut

I am testing some detailed input handling and the tests work as expected from both the editor and the command-line without the --headless option. However, these fail when headless, probably for the reasons explained in comments in godotengine/godot#73557 .

Accepting this engine behavior, I'm now looking for:

GUT's recommended way to skip tests--I didn't find anything in doc
A (straightforward) way to detect headless mode to trigger the skips

I found @russmatney's comment in #360 useful for point 2, but was hoping in the meantime someone had found a better way.

TIA!

Ah, found an answer to point 1 in https://gut.readthedocs.io/en/latest/New-For-Godot-4.html#what-s-new-changed-in-gut-9-0-0-for-godot-4-0 and sorry for not seeing it before.

Added support for a skip_script test-script variable. This can be added to any test-script or inner-class causing GUT to skip running tests in that script. The script will be included in the “risky” count and appear in the summary as skipped. This was done to help porting tests to 4.0 but might stick around as a permanent feature.

var skip_script = 'The reason for skipping. This will be printed in the output.'

Well, I was just impatient. Below approach seems to work well enough and this could be just a documentation issue.

extends GutTest

var skip_script

func _init():
    if DisplayServer.get_name() == "headless":
        skip_script = "Skip Input tests when running headless"

Unfortunately, I think GUT only looks to see if the var exists, not the value. Which means you have to comment out all that code if you want to run those tests.

Adding a check to see if it has a value and only skipping when not null would be a solution, but a more concrete structure would be better.

One approach might be to put all your input tests in a different folder and make another gut config to use for headless which does not include that folder. You can specify the gut_config file to use in the command line.

Gut/addons/gut/gut.gd

Line 706 in a627250

if(skip_script != null):

appears to be the only check. I can also use the separate gut_config approach, thanks!

Yea, that's the only one. I thought I was doing a has_property kind of thing, but I'm doing a .get("skip_script") and checking for a null value (which is what it is if the property does not exist). Having a more formal solution would be nice, but maybe this can just be documented.

This isn't how I envisioned this feature working but I really like using it that way. The only potential issue I see is using
_init to set the value. Right now the GutTest class is not doing anything in _init, but it may in the future. I'd rather find a way to do this where I don't have to document that you should call super's _init; but then everything works if you don't...until GUT changes and weird issues start to popup. Not to mention that super's _init will still be called in Godot 3.0 but not in 4.0 (iirc).

So maybe the GUT should check for skip_script after calling before_all. That way we don't encourage people to override _init which may cause future issues.

My original idea with var skip_script was for temporarily skipping scripts (initially those that were not yet fully converted to Godot 4). It was a quick hack I threw together. I chose a variable so that it could seen easily at the top of the script. Using it to conditionally skip scripts is not in the spirit of the variable. It also requires two steps, declaring it and then setting it.

I never really liked how "magic" it was. "Oh, I just declare a variable and then something different happens?...ok". I probably shouldn't have documented its existence, heh.

I think a more formal solution for conditionally skipping scripts would be to create a virtual should_skip_script() method in GutTest. You can then override this method in your test script to conditionally skip a script.

Your implementation should return true or a string to skip the script.
If a string is returned then it will be used in the message that GUT prints when skipping the script.
Return false to run the script. Anything else will generate an error message but still run the script.
GutTest should have an implementation that just returns false.
After this is implemented the var skip_script approach should be deprecated and removed in the next release.
When should_skip_script results in the script being run, it should not cause any orphans.
Skipped scripts are included in the risky count (for a lack of any better place).

Additional benefits of this approach

Having a method would allow you to clean up any orphans you might create in variable declarations that you would free in after_all.
You can still put should_skip_script at the top of the script where it is more visible.
Later we could add a should_skip_test that passes in the test name which would allow you to conditionally skip specific tests (this doesn't sound like a great idea, but maybe it is).

If opting for a bit more design, I wonder whether there's one approach that can work for both complete scripts and for individual inner classes.

We could add a should_skip_file which would skip the script and any inner classes, so no tests run in the entire file.

You could also make your own script class that extends GutTest that has should_skip_script implemented and extend that instead of GutTest to "auto skip" a script/inner.

This has been implemented in main. See CHANGES.md for info on should_skip_script virtual method.

Question: How to skip Input tests which fail under --headless?