Giters

tebeka / go2xunit

Convert "go test" output to xunit compatible (used in Jenkins/Hudson)

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Parser cannot handle tests run with t.Parallel()

Redundancy opened this issue · comments

commented

When running tests in parallel, the test result may not immediately follow the RUN line:

=== RUN TestA
=== RUN TestB
=== RUN TestC
--- PASS: TestA (0.00 seconds)
--- PASS: TestB (0.00 seconds)
--- PASS: TestC (0.00 seconds)

I may try and fix this in my fork and send you a patch.

commented

How can you tell which output belongs to which test?

To quote the zen of Python - "In the face of ambiguity, refuse the temptation to guess.".

This should be fixed in the go test utility, see this issue for example.

commented

I guess the question would be - if we can't reliably determine the correct test for any given output, would it be better to fail to interpret the results entirely, or to throw away the ambiguous information?

We could track the number of in-flight tests unambiguously based on the start/stop messages, and for the unambiguous cases, just put it in there. Iff there's any ambiguity, we can mark those tests as ambiguous, emit a line to the console explaining the issue, and potentially replace the output from the test with a marker string of some sort so that the test maintainer can solve the issue.

I'd argue that the output capture is less important than the correct interpretation of the test results in a lot of the use-cases that require you to convert to xunit output?

commented

I'm a firm believer in the "fail fast" methodology and not doing GIGO.

Since what the user will see is only the Jenkins web page with the results - they'll have no idea if some of the tests we ignored.

and for the unambiguous cases, just put it in there

Can you elaborate? I don't think there's a bullet proof unambiguous cases.

I'd argue that the output capture is less important than the correct interpretation of the test results in a lot of the use-cases that require you to convert to xunit output?

I don't follow. If "correct interpretation" is what's important - than there's nothing we can do (IMO).

Maybe there users can provide a command line option to go2xunit telling it it's parallel tests and try to do you best. And we state that in this case - GIGO might follow.