Incorrect counting of tests

[Jesse Alama]

(Working with 7.5, CS variant, on macOS. I'm not sure about the regular variant on 7.5.)

Consider this module (let's put it in the file foo.rkt):

#lang racket/base

(module+ test
  (require rackunit)
  (test-case
      "whatever"
    (check-equal? 1 2)))

If you do raco test foo.rkt, here's what you get:

raco test: (submod "unit.rkt" test)
--------------------
whatever
FAILURE
name:       check-equal?
location:   unit.rkt:7:4
actual:     1
expected:   2
--------------------
1/2 test failures

Expected output: 1/1 test failure.

Curiously, if you change the test to a passing one, e.g., by replacing (check-equal? 1 2) with (check-equal? 1 1), the output becomes

raco test: (submod "unit.rkt" test)
1 test passed

which suggests the the counting of tests is correct if all tests pass, but wrong if at least one test fails.

[Jesse Alama]

The same error can be observed in 7.3, regular variant, on FreeBSD.

[Jack Firth]

I can confirm I see the same problem. Can be reproduced without involving raco test by inspecting the test log directly. This program:

#lang racket

(require rackunit
         rackunit/log)

(test-case "whatever"
  (check-equal? 1 2))

(test-log)

...outputs a test failure and '(1 . 2) (that's the number of failed tests and the number of total tests, respectively) when run. So it's definitely something wrong in rackunit, not raco test.

[Jesse Alama]

Thanks! Do you think this will make it into 7.6?

[Jack Firth]

I think so. The merge window ends on on the 15th, and there isn't yet any entry in the GitHub Releases of this repo for 7.6. So it should be included.