Problems parsing XML created with 0.3.pre3
sidonath opened this issue · comments
Hi rspec_junit_formatter for a while on a one of our projects to format the output of specs into something readable by Jenkins. We started a fresh project recently and ran into an old issue that we hacked around previously.
We run into problems when we enable colored output in RSpec — we want colors to show in Jenkins console output, but if ANSI escape codes end up in the XML it renders the XML unreadable to Jenkins. I've seen the freshest versions of the gem and thought it would address the issues cleanly, however, Jenkins still complains about invalid XML. As a matter of fact, even Safari and Chrome complain about the structure of the XML with error messages similar to:
error on line 2 at column 14: xmlParseCharRef: invalid xmlChar value 27
The characters are indeed escaped correctly (i.e. \u1b is correctly escaped as ), but even the escaped character is not valid in XML. So I think for some classes of characters, the best solution is to simply remove them from the output.
For instance, when you save the output of the example spec to a file and try to open with Safari or Chrome, I get the following render:
I'll gladly help with a pull request, but I'm not sure what's the best way to proceed. Initially I wanted to remove only ANSI escape codes, but apparently it's a broader issue and maybe some characters should simply be removed instead of trying to escape them.
Thanks for the wonderful feedback!
Yeah, XML is pesky here because while it might be technically capable of representing control characters using character entities, the XML 1.0 standard says they are forbidden. XML 1.1 actually amended this to include control characters, basically any character except NULL (#0). But I see the parser above choked on a NULL. Some parsers just carry on when encountering unexpected characters, but others seem to choke. And nobody seems to have adopted XML 1.1.
It seems I have been defeated by XML. I've updated master to omit characters outside the character set defined in XML 1.0, and to escape the well-known entities and anything listed as "discouraged" in XML 1.0.
Please try 0.3.0.pre4 and let me know if it fixes things up for you.
Maybe illegal characters should be replace by an escape code or something? Hmm.
Maybe something like what String#dump does.
I've just pushed 0.3.0.pre5 (what a productive day!) which replaces illegal characters with ruby-like escapes. Give them both a go and let me know what you think. :-)
@sj26 gave them both a try and both work great, thanks!
I still feel a need to remove ANSI escape codes specifically, to clean up the output:
Do you think it makes sense to add that logic here? It's quite easy to override this method in my own code as well.
Interesting. Those codes are produced by the rspec base text formatter doing it's colorizing thing, but that should be turned off or stripped in this formatter. I'd consider your output a bug.
What version of rspec are you using? (bundle info rspec-core)
@sj26 hah, funny how I never stopped to think about it that way. Yeah, you're right, this was RSpec issue that was fixed in 3.6 (we were using 3.5). After the upgrade this doesn't seem to be a problem anymore
Oh dear! I just tried using 3.5.0 and 3.5.4 and I still couldn't replicate what you were seeing I'm afraid. Glad an upgrade made it go away, though!
Sounds like the problems solved so I'll close this ticket. v0.3.0 should be out for real, soon!
Actually, we still have a problem with Jenkins, but that's definitely not an issue here.
Thanks for the help!
@sj26 actually, I made a mistake in my test earlier and it turned out that the RSpec upgrade didn't fix the problem. Moreover, looking at this RSpec issue, this seems as the desired behavior.
This is the simplest way for me to reproduce the issue locally:
test.rb:
RSpec.describe 'Dummy' do
specify { expect(foo: 1, bar: 2).to eq(foo: 2, bar: 2) }
end$ rspec -v
RSpec 3.6
- rspec-core 3.6.0
- rspec-expectations 3.6.0
- rspec-mocks 3.6.0
- rspec-rails 3.6.0
- rspec-support 3.6.0
$ rspec --format RspecJunitFormatter test.rb --out test.xml
$ cat test.xml
<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="rspec" tests="1" failures="1" errors="0" time="0.025031" timestamp="2017-06-15T01:00:52+09:00">
<properties>
<property name="seed" value="10906"/>
</properties>
<testcase classname="test" name="Dummy should eq {:foo=>2, :bar=>2}" file="./test.rb" time="0.023363"><failure message="
expected: {:foo=>2, :bar=>2}
got: {:foo=>1, :bar=>2}
(compared using ==)
Diff:\e[0m
\e[0m\e[34m@@ -1,3 +1,3 @@
\e[0m\e[0m :bar => 2,
\e[0m\e[31m-:foo => 2,
\e[0m\e[32m+:foo => 1,
\e[0m" type="RSpec::Expectations::ExpectationNotMetError">Failure/Error: \e[0m\e[32mspecify\e[0m { \e[32mexpect\e[0m(\e[35mfoo\e[0m: \e[1;34m1\e[0m, \e[35mbar\e[0m: \e[1;34m2\e[0m).to eq(\e[35mfoo\e[0m: \e[1;34m2\e[0m, \e[35mbar\e[0m: \e[1;34m2\e[0m) }
expected: {:foo=>2, :bar=>2}
got: {:foo=>1, :bar=>2}
(compared using ==)
Diff:\e[0m
\e[0m\e[34m@@ -1,3 +1,3 @@
\e[0m\e[0m :bar => 2,
\e[0m\e[31m-:foo => 2,
\e[0m\e[32m+:foo => 1,
\e[0m
./test.rb:2:in `block (2 levels) in <top (required)>'</failure></testcase>
</testsuite>
This is my fix for now:
class ColorlessRSpecJUnitFormatter < RSpecJUnitFormatter
RSpec::Core::Formatters.register self, :start, :stop, :dump_summary
ANSI_ESCAPE_CODE = /\e\[[\d;]+[[:alpha:]]/
private
def escape(text)
super(text.to_s.encode(Encoding::UTF_8).gsub(ANSI_ESCAPE_CODE, ''))
end
endOhhh, I see. It's colourising the diff which is why I couldn't replicate it. The normal formatter output is correctly without colour, but the diff output has been coloured. I think that might be a bug in rspec. This formatter explicitly asks for the non-colourised output but is receiving colourised details. Looks like it's using a global instead of the local colouriser.
Will investigate.
I looked through the source of RSpec core, sprinkling it with debugging output and I couldn't see that there's a concept of a local colorizer. From what I've seen it uses the global settings regardless of where the output goes (as suggested by the RSpec issue I linked previously).
The base text formatter calls #fully_formatted on a bunch of the notifications. These methods take a colorizer which is RSpec::Core::Formatters::ConsoleCodes by default which decorates things with ANSI codes. In the JUnit formatter I'm calling the non fully-formatted versions of these which call other methods which use a NullColorizer to avoid the codes. So for the most part the output isn't colorized, and this is independent of the color options on the command line.
The diff, however, is appended to the message and is calculated against a differ constructed with the global color argument. So by the time this message hits the formatters it has already had the coloring baked into it based on the command line arguments.
I could strip ANSI codes from the message but would have no way to determine if they're a useful part of your error output or not. For example, you might be building a library for colouring terminal output which has expectations on escape sequences.
This is a tricky one.
Oh, right, I didn't consider that case
Since one can't pass options to formatter, the only option seems to provide two formatters and document their difference
@chrishough please try the released v0.3.0, it should also strip ansi escapes from diff output.
That worked as expected @sj26, thank you