RSpec incremental summarized output
danielwaterworth opened this issue · comments
I'm one of the developers working on Discourse and we have recently starting using parallel_tests. I've been tasked with making parallel rspec work more like regular rspec. We'd like to see progress as it happens and then see a single summary at the end (much like as is described in #411).
The way I've chosen to go about doing this is to make each subprocess use a custom rspec formatter that produces a json object for each event (code). The main process then parses these events and does its own reporting (code). See this PR on discourse for more information.
This will work for us, but judging by some of the issues in this tracker (#411, #523, #704), there are other users of this gem that might also benefit from such changes. However, it's quite a substantial departure from the current behavior.
Would you be interested in receiving PRs that add this functionality?
I like the current simplicity of "run these commands and display their output", that seems very sturdy and won't break between different frameworks and their versions.
Having an opt-in formatter that produce a more standardized output (that can later be expanded to all test frameworks) and a parser for that seems like a good enhancement though.
The common format already exists with junit output https://github.com/sj26/rspec_junit_formatter + https://github.com/aespinosa/minitest-junit + https://github.com/cucumber/cucumber-ruby/blob/master/lib/cucumber/formatter/junit.rb so ideally reuse that / instruct users what gems to add and add --junit flag to parse the the resulting files (might have to add the --format or so flags since we need to set known output locations, not sure).
It looks like we're in slightly different positions here:
Discourse uses one version of one testing framework and we decide when to upgrade it, but parallel_tests supports many versions of many testing frameworks and can't anticipate new versions.
In light of that, I definitely see the benefit of a common test suite format. Any summarizing and displaying becomes test-suite independent and it reduces the amount of test-framework specific code compared to my approach.
However, for our purposes, junit xml is less than ideal because it's not incremental, but it looks like TAP-Y/J might work. There's https://github.com/rubyworks/rspec-ontap and https://github.com/rubyworks/minitap, but cucumber is missing.
I would be really interested if this gets rolled into parallel_tests, very cool work