Provide a way to suppress excessive output/stacktraces
ScottG489 opened this issue · comments
Testing Problem
Since jqwik is running tests many times, it has the possibility to produce a lot of output. This has particularly been a problem in tests that expect an error to occur and log a message with a stacktrace for every run.
Perhaps it is outside the scope of responsibilities of this library, but it may make sense for jqwik to provide a solution.
Suggested Solution
One idea is that jqwik could automatically intercept output and deduplicate it. For instance, if all of the exceptions logged are exactly the same, there's no point in printing them all. Rather we could print each once, along with a mention of how many times the exception occurred. This could also possibly be controllable via a param to the @Property annotation.
Discussion
Let me know your thoughts or if you have ideas for how this could be cleanly implemented outside of this library.
@ScottG489 Thanks for bringing this up.
I'm also not a fan of output pollution by tests. "Excessive" is in the eye of the beholder, though, so a concrete example of what part of output or stack traces you'd like to have suppressed would be welcome.
I think the problem can be more generally stated as deduplicating any output, not just a stacktrace.
So jqwik could look at the output for each test run and compare it to the output from previous runs and if it's the same, suppress it and possibly increment a counter to let the user know how many times some given output was repeated.
I'm not sure if you'd have the context to be able to capture all output though? Like loggers I think you might be able to hook into, but stderr or stdout?
I think I understand what you mean on an abstract level. Still, an example would come in handy ;-)
For sure! Not sure if this would be what you're looking for, but if you look at the bulk of the logs for a build on one of my projects, you can see a bunch of stacktraces logged:
https://github.com/ScottG489/conjob/actions/runs/5446904729/jobs/9908291384
Is this helpful?
Looking at the logs from this job I see a few different aspects. Some of them are:
- You can already suppress jqwik reporting for successful properties and examples via
jqwik.reporting.onlyfailures=true - Most (I didn't count just my guess ;-) of the stack traces are logged by a 3rd party component, e.g. conjob.core.job.JobRunner and should be suppressable by configuring the log framework.
- Deduplicating console output across tries: This sounds like a task for another library, at least the part of intercepting the different ways that logs can get to stdout and stderr.
- Stack traces that are part of standard failure reporting should only occur twice anyway: Once for the original failing sample and a second time for the shrunk sample.
Can you see a low-hanging fruit to get the whole idea started?
Thanks for taking a look!
- You can already suppress jqwik reporting for successful properties and examples via jqwik.reporting.onlyfailures=true
I think I may do this, but since they only print out once per test, it isn't too big of a deal either way.
- Most (I didn't count just my guess ;-) of the stack traces are logged by a 3rd party component, e.g. conjob.core.job.JobRunner and should be suppressable by configuring the log framework.
- Deduplicating console output across tries: This sounds like a task for another library, at least the part of intercepting the different ways that logs can get to stdout and stderr.
I think this is kinda where I arrived after posting and thinking about this more as well. I also think while it would be nice to deduplicate the logs, I wonder if I'm misplacing my focus and if I should care about the output anyways. I think I may just suppress all output and if this makes it difficult to troubleshoot failures I'll revisit.
I'll let you decide if you want to close this.
There's one more thing that could be useful to you. In the testing module you can find the @SuppressLogging annotation that I use to suppress expected jqwik logging (mostly warnings) in tests.
The implementation mostly works but it's anything but polished:
https://github.com/jqwik-team/jqwik/blob/main/testing/src/main/java/net/jqwik/testing/SuppressLogging.java
Feel free to use it directly or copy and modify it.
Closing for now. Feel free to re-open if you encounter a concrete feature request.
I'm not seeing @SuppressLogging anywhere in the published artifact so I can use it directly. Is that expected?
I'm on the latest version.
you'll have to add a dependency to "jqwik-testing"
I added this to my build.gradle but the dep doesn't seem to be getting pulled in:
testImplementation "net.jqwik:jqwik-testing:1.7.4"
Strange. I've just tried it in a new project and @net.jqwik.testing.SuppressLogging is available for import.
Only for testing code, but that's intended by testImplementation. Any error message or warning?
It must be something on my end. It doesn't even seem to be pulling down the dep. I don't see the jar listed under my external dependencies.
Maybe you’re working on a Maven mirror that doesn’t pull everything?
I don't think so. I only have maven central specified:
repositories {
mavenCentral()
}You can see my full configuration here (minus the testing dep atm):
https://github.com/ScottG489/conjob/blob/master/build.gradle
https://github.com/ScottG489/conjob/blob/master/gradle/unit-test.gradle
I added unitTestImplementation "net.jqwik:jqwik-testing:1.7.4" under line 40. I also aligned the versions.
Everything seems to be available on maven central: https://repo.maven.apache.org/maven2/net/jqwik/jqwik-testing/1.7.4/
Are the other 1.7.4 artefacts being picked up?