What is the best way to make integration tests with GTK?

[Arkadiusz Piekarz]

Hi, do you know what is the preferred strategy to make integration tests with GTK? The problem with it is there is a global resource that tests may share - the GTK library instance. It manages the lifetime of widgets / GObjects by itself, requires access from only one thread and AFAIK cannot be deinitialized to clean up resources.

I explored the following options while using gtk-test crate:

1) Put each test in a separate file in the "tests" directory.

Pros:
+ tests are isolated
+ automatic registration of tests
+ quick compile times
Cons:
- every file is in the same folder, it's hard to distinguish categories of them
- slower execution than option 2

2) In the "tests" folder create mod.rs with names of subfolders, create subfolders with their own mod.rss, put tests in separate files in subfolders.

Pros:
+ categorization of tests in the form of subfolders
+ quick compile times
+ quick test execution
+ less boilerplate than option 5
Cons:
- tests are NOT isolated! they use the same GTK library instance and can access windows/application from other tests
- tests cannot in parallel because of GTK limitations
- no automatic registration of tests (unless you use automod crate, but then IDEs claim the tests are not part of the project and jumping to definitions stops working)

3) Using option 2 with rusty-fork crate.

Pros:
+ tests are isolated
Cons:
- execution of tests continues even when one test failed
- lots of noise in backtraces
- color-backtrace crate seems to not work with rusty-fork backtraces
- slower execution than option 2

4) Using option 2 with locking a global mutex at the start of every test.

Pros:
+ better isolation, but not full
Cons:
- the next test can see still living windows from the previous tests, not being deleted by GTK even after closing them (you can unsafely destroy them manually at the end of tests, but then GTK prints critical errors from its assertions)
- developer has to remember to lock the mutex
- execution becomes similar in speed to single-threaded

5) Create a separate "integration_tests" folder with subfolders, where each is a crate with it's own Cargo.toml, src and tests folders, then put tests in the last ones; add "integration_tests" to workspace in the main project's Cargo.toml to share Cargo.lock with the production code.

Pros:
+ tests are isolated
+ categories are visible in the form of subcrates
+ automatic registration of tests
Cons:
- compile times are 2x slower
- lots of boilerplate with the additional folders and files
- subcrates need to state their dependencies in their Cargo.tomls, which leads to duplication and maintenance burden
- slower execution than option 2

As you can see every solution I tried had issues. Do you know if it could be done in a better way?
Thanks in advance.

[Sebastian Dröge]

@GuillaumeGomez or @antoyo might have an opinion here, as authors of the gtk-test crate :)

[antoyo]

Err, you have explored much more options than I did :) .

One thing you could try, if you're on Linux, is to use xvfb: maybe that will allow you to run the tests in parallel if the window in every instance of xvfb can have the focus at the same time (I'm really not sure about this, you'll have to check if that is really the case).

[Arkadiusz Piekarz]

Thanks for the xvfb tip, I like how it doesn't draw windows and doesn't add entries to the taskbar during test execution, although it adds a few seconds due to its initialization. I will experiment with it more.

Assuming I would like to explore running multiple tests on the same GTK library instance, do you have any advice how I could safely destroy all windows at the end of a given test? Or otherwise "deinitialize" the GTK library? I already tried invoking close() on all top level windows and running event loop until there are no more pending events, but some GtkTooltipWindows still linger after that.

[Arkadiusz Piekarz]

Nevermind, after more research I went with option 3, meaning using the "tests" folder with rusty-fork crate. Updated status of problems:

• "execution of tests continues even when one test failed" - still stands
• "lots of noise in backtraces" - still prints two stacktraces instead of one, but IntelliJ automatically detects and folds them
• "color-backtrace crate seems to not work with rusty-fork backtraces" - actually it works for the stacktrace of the test "child-process" (not stacktrace of the rusty-fork "parent-process") and can be forced to always output colors
• "slower execution than option 2" - it's not a sensible comparison, because the option 2 suffers from race conditions; I got the option 3 with almost 60 integration tests down to 3 seconds thanks to rusty-fork allowing multithreaded execution

Xvfb on its own doesn't seem to solve the problems of multithreading if we're using just one instance of it and have tests sharing GTK library, but it's good for hiding the windows.

Thank you for your answers.