Double-count issue when using two reporters
AndrewGnagy opened this issue · comments
I'm attempting to use a custom reporter to report results to both junit and the pretty-printer.
What I've found is that this will double-count the number of assertions and failures. For example:
Ran 5 tests in 6.059 seconds
10 assertions, 2 failures, 2 errors.
(Should be 1 assertion per test for 5 total. 1 failure and 1 error.)
A work-around suggested by @miikka is binding clojure.test/*report-counters* to nil for one of the reports:
(defn report [m]
(pretty/report m)
(binding [clojure.test/*report-counters* nil]
(xml-report m)))
Which works.
Is there a more proper way of handling this?
Because of the way clojure.test
works, that might be the best solution. We could add in something that combines multiple reporters automatically, or potentially we could try and create a new reporter abstraction we could layer on top of clojure.test/report
, but adding new abstractions is something I tend to be cautious about.
I'm using following function in boot-alt-test to combine collection of reporters, should we add this to eftest.report
?
(defn combined-reporter
"Combines the reporters by running first one directly,
and others with clojure.test/*report-counters* bound to nil."
[report & rst]
(fn [m]
(report m)
(doseq [report rst]
(binding [clojure.test/*report-counters* nil]
(report m)))))
FWIW, I just wrote my own version of such a function, and I’m not encountering the double counting.
Update: Ah, never mind, I didn’t read the TP closely enough. I am indeed seeing the same issue with my function.
My function
(defn- multi-report
"Accepts n reporting functions, returns a reporting function that will call
them all for their side effects and return nil. I tried to just use juxt but
it didn’t work. Maybe because some of the reporting functions provided by
eftest are multimethods, I don’t know."
[& fs]
(fn [event]
(doseq [f fs]
(f event))))
Ah, never mind, I didn’t read the TP closely enough. I am indeed seeing the same issue with my function.
i'm upvoting the need for a duel reporter --> i wanna be able to run lein eftest in my CI that produces a junit report while outputing to console in development.
(or just use both)