Junit output does not seem valid (parsing error in Jenkins)

Question

Junit output does not seem valid (parsing error in Jenkins)

dduportal opened this issue 3 years ago · comments

Hello there 👋

First of all thanks for this really useful tool!

While trying the junit output feature from the latest v1.10.0, I was not able to feed it to Jenkins as a Junit standard report.
The error happens during the Junit parsing:

java.lang.NullPointerException
	at hudson.tasks.junit.CaseResult.getPackageName(CaseResult.java:399)
	at hudson.tasks.junit.TestResult.tally(TestResult.java:795)
	at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:145)
	at hudson.FilePath.act(FilePath.java:1161)
	at hudson.FilePath.act(FilePath.java:1144)
	at hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:107)
	at hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:149)
	at hudson.tasks.junit.JUnitResultArchiver.parseAndSummarize(JUnitResultArchiver.java:243)
	at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:63)
	at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:29)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

It looks like that the XML generated by CST. is not valid:

<testsuites failures="0" tests="14" time="0"><testsuite><testcase name="File Content Test: image user created as non-root user" time="0"></testcase><testcase name="File Content Test: subuid contains user with 65,536 subordinate UIDs" time="0"></testcase><testcase name="File Content Test: subgid contains user 65,536 subordinate GIDs" time="0"></testcase><testcase name="File Existence Test: Google Container Test CLI" time="0"></testcase><testcase name="File Existence Test: HADOLINT" time="0"></testcase><testcase name="File Existence Test: Root" time="0"></testcase><testcase name="File Existence Test: img" time="0"></testcase><testcase name="File Existence Test: img newuidmap" time="0"></testcase><testcase name="File Existence Test: img newgidmap" time="0"></testcase><testcase name="File Existence Test: subuid" time="0"></testcase><testcase name="File Existence Test: subgid" time="0"></testcase><testcase name="File Existence Test: user directory" time="0"></testcase><testcase name="File Existence Test: home user" time="0"></testcase><testcase name="Metadata Test" time="0"></testcase></testsuite></testsuites>

(The cst.yml file is public and available here: https://github.com/jenkins-infra/docker-builder/blob/main/cst.yml ).

While iterating from an existing working Junit report from one of my Java projects, I was able to correctly feed this modified XML report to Jenkins:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
  xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report-3.0.xsd" 
  version="3.0" name="report" failures="0" tests="14" time="0" errors="0" skipped="0">
  <testcase name="File Content Test: image user created as non-root user" time="0"></testcase>
  <testcase name="File Content Test: subuid contains user with 65,536 subordinate UIDs" time="0"></testcase>
  <testcase name="File Content Test: subgid contains user 65,536 subordinate GIDs" time="0"></testcase>
  <testcase name="File Existence Test: Google Container Test CLI" time="0"></testcase>
  <testcase name="File Existence Test: HADOLINT" time="0"></testcase>
  <testcase name="File Existence Test: Root" time="0"></testcase>
  <testcase name="File Existence Test: img" time="0"></testcase>
  <testcase name="File Existence Test: img newuidmap" time="0"></testcase>
  <testcase name="File Existence Test: img newgidmap" time="0"></testcase>
  <testcase name="File Existence Test: subuid" time="0"></testcase>
  <testcase name="File Existence Test: subgid" time="0"></testcase>
  <testcase name="File Existence Test: user directory" time="0"></testcase>
  <testcase name="File Existence Test: home user" time="0"></testcase>
  <testcase name="Metadata Test" time="0"></testcase>
</testsuite>

It includes the following changes (which might be a cause or not, I did not dig enough in the details so we might be able to remove some of these changes and still have a valid XML report):

Adding an <?xml version="1.0" encoding="UTF-8"?> first line
Switching from the singular form testcases to a singular form testcase
Adding a DTD grammar to the testcase markup
Following the grammar validation, adding 2 missing attributes to the testcase markup: errors and skipped

I'm not really sure where to search for the correct way of validating a Junit XML report, and I did not tried (yet) another tool (as Gitlab or Github Action, or even a Maven surefire report) to parse the Junit report, but I wanted to report this issue to start the discussion.

If you want to reproduce the "Jenkins" case I have, you can use https://github.com/dduportal/jenkins-k8s (which is public) to spawn a Jenkins instance with all the necessary plugins and setup. At the time of writing this comment, I was using the commit https://github.com/dduportal/jenkins-k8s/tree/74ddada3b24ec34fec20c5810a9496443087f3f7 which means Jenkins 2.263.2 and Junit's Jenkins plugin 1.48.

Deleted user · Answer 1 · Tue Jan 26 2021 23:51:22 GMT+0800 (China Standard Time)

Hello! First of all, thank you for the tool, it does its job well and is written very concisely.

I'm also trying to use JUnit-formatted reports in Jenkins and it fails with the same error message.
It would be awesome if JUnit reports are Jenkins-compatible.

Aaron Turner · Answer 2 · Wed Jan 19 2022 21:16:26 GMT+0800 (China Standard Time)

Also having the same issue, when using the xunit plugin in jenkins.

xunit plugin (supports JUnit format) parsing JUnit xml out of container-structure-test.

ARNING: The file '/home/ubuntu/workspace/always.xml' is an invalid file.
WARNING: At line 1 of file:/home/ubuntu/workspace/always.xml:cvc-complex-type.4: Attribute 'name' must appear on element 'testsuite'.
WARNING: At line 1 of file:/home/ubuntu/workspace/always.xml:cvc-complex-type.4: Attribute 'tests' must appear on element 'testsuite'.
WARNING: At line 1 of file:/home/ubuntu/workspace/always.xml:cvc-complex-type.4: Attribute 'failures' must appear on element 'testsuite'.
WARNING: At line 1 of file:/home/ubuntu/workspace/always.xml:cvc-complex-type.4: Attribute 'errors' must appear on element 'testsuite'.

Output form container-structure-test:

<testsuites failures="1" tests="1" time="0">
  <testsuite>
    <testcase name="Metadata Test" time="0">
      <failure>label com.example.vendor not found in image metadata</failure>
      <failure>label com.example.owner not found in image metadata</failure>
      <failure>label com.example.project.name not found in image metadata</failure>
      <failure>label com.example.project.type not found in image metadata</failure>
      <failure>label com.example.service.name not found in image metadata</failure>
      <failure>label com.example.service.type not found in image metadata</failure>
      <failure>label com.example.image.description not found in image metadata</failure>
      <failure>label com.example.image.version not found in image metadata</failure>
      <failure>label com.example.image.build not found in image metadata</failure>
      <failure>label com.example.image.licenses not found in image metadata</failure>
      <failure>label com.example.image.created not found in image metadata</failure>
    </testcase>
  </testsuite>
</testsuites>

@dduportal did you ever find a work around for this in Jenkins?

0b7702e2 · Answer 3 · Thu Jan 20 2022 09:58:15 GMT+0800 (China Standard Time)

I had the exact same issue as @dduportal while using junit with Jenkins Pipeline. After some trial and error, I found adding a "name" attribute to the "testsuite" (singular) element gets rid of the problem. Something like
<testsuite name="MyPackageName.MyClassName">

Thomas Cooper · Answer 4 · Sun Sep 11 2022 10:41:02 GMT+0800 (China Standard Time)

I started hacking on this today. The workaround from @b7702e2 produces the best result (with hardcoded name="container-structure-test.test"). Tested on a fresh Jenkins 2.361.1 install with latest JUnit plugin (1119.1121.vc43d0fc45561).

$ ../container-structure-test/out/container-structure-test test --image test-image --config test.yml -o junit
<?xml version="1.0" encoding="UTF-8"?><testsuites failures="0" tests="2" time="1.57702953"><testsuite name="container-structure-test.test"><testcase name="Command Test: whoami" time="1.57702953"></testcase><testcase name="Metadata Test" time="0"></testcase></testsuite></testsuites>

@dduportal as an FYI, I experimented with rendering out an report based on the Surefire schema mentioned in the original post but it does not render correctly at all in Jenkins. Tests are not viewable and a default "(root)" package is displayed in the UI. I would love to produce a schema-conforming document but one doesn't seem to exist for JUnit itself. Many plugins are producing their own format such as maven-surefire. See below screenshot:

Since a few folks have already mentioned trying to get this to work on Jenkins, suggest we merge in this kind of fix and add any other formats as separate --output types.