Choking on interpolation with heredoc
akimd opened this issue · comments
Hi!
We fell on a case on which unparser chokes. parser manages to read this properly. It appears to be quite a minimal example: if you remove the if, or the return, or the interpolation, then it works.
$ unparser --verbose -e 'if true
return <<~EOF
#{42}
EOF
end
'
(string)
Original-Source:
if true
return <<~EOF
#{42}
EOF
end
Generated-Source:
if true
return <<-HEREDOC
end
Original-Node:
(if
(true)
(return
(dstr
(begin
(int 42))
(str "\n"))) nil)
Generated-Node:
#<Parser::SyntaxError: unterminated string meets end of file>
/opt/local/lib/ruby3.0/gems/3.0.0/gems/parser-3.0.0.0/lib/parser/diagnostic/engine.rb:72:in `process'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/parser-3.0.0.0/lib/parser/lexer.rb:23669:in `diagnostic'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/parser-3.0.0.0/lib/parser/lexer.rb:11539:in `advance'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/parser-3.0.0.0/lib/parser/base.rb:252:in `next_token'
(eval):3:in `_racc_do_parse_c'
(eval):3:in `do_parse'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/parser-3.0.0.0/lib/parser/base.rb:190:in `parse'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/unparser-0.5.6/lib/unparser.rb:98:in `block in parse_either'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/mprelude-0.1.0/lib/mprelude.rb:85:in `wrap_error'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/unparser-0.5.6/lib/unparser.rb:97:in `parse_either'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/mprelude-0.1.0/lib/mprelude.rb:168:in `bind'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/unparser-0.5.6/lib/unparser/validation.rb:63:in `from_string'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/unparser-0.5.6/lib/unparser/cli.rb:40:in `validation'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/unparser-0.5.6/lib/unparser/cli.rb:143:in `public_send'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/unparser-0.5.6/lib/unparser/cli.rb:143:in `process_target'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/unparser-0.5.6/lib/unparser/cli.rb:133:in `block in exit_status'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/unparser-0.5.6/lib/unparser/cli.rb:132:in `each'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/unparser-0.5.6/lib/unparser/cli.rb:132:in `exit_status'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/unparser-0.5.6/lib/unparser/cli.rb:64:in `run'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/unparser-0.5.6/bin/unparser:10:in `<top (required)>'
Error: (string)
I can reproduce without heredoc, but the \n after #{42} is required.
$ unparser --verbose -e 'if true
return "#{42}"
end
'
(string)
Original-Source:
if true
return "#{42}"
end
Generated-Source:
if true
return "#{42}"
end
Original-Node:
(if
(true)
(return
(dstr
(begin
(int 42)))) nil)
Generated-Node:
(if
(true)
(return
(dstr
(begin
(int 42)))) nil)
Success: (string)
$ unparser --verbose -e 'if true
return "#{42}
"
end
'
(string)
Original-Source:
if true
return "#{42}
"
end
Generated-Source:
if true
return <<-HEREDOC
end
Original-Node:
(if
(true)
(return
(dstr
(begin
(int 42))
(str "\n"))) nil)
Generated-Node:
#<Parser::SyntaxError: unterminated string meets end of file>
/opt/local/lib/ruby3.0/gems/3.0.0/gems/parser-3.0.0.0/lib/parser/diagnostic/engine.rb:72:in `process'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/parser-3.0.0.0/lib/parser/lexer.rb:23669:in `diagnostic'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/parser-3.0.0.0/lib/parser/lexer.rb:11539:in `advance'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/parser-3.0.0.0/lib/parser/base.rb:252:in `next_token'
(eval):3:in `_racc_do_parse_c'
(eval):3:in `do_parse'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/parser-3.0.0.0/lib/parser/base.rb:190:in `parse'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/unparser-0.5.6/lib/unparser.rb:98:in `block in parse_either'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/mprelude-0.1.0/lib/mprelude.rb:85:in `wrap_error'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/unparser-0.5.6/lib/unparser.rb:97:in `parse_either'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/mprelude-0.1.0/lib/mprelude.rb:168:in `bind'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/unparser-0.5.6/lib/unparser/validation.rb:63:in `from_string'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/unparser-0.5.6/lib/unparser/cli.rb:40:in `validation'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/unparser-0.5.6/lib/unparser/cli.rb:143:in `public_send'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/unparser-0.5.6/lib/unparser/cli.rb:143:in `process_target'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/unparser-0.5.6/lib/unparser/cli.rb:133:in `block in exit_status'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/unparser-0.5.6/lib/unparser/cli.rb:132:in `each'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/unparser-0.5.6/lib/unparser/cli.rb:132:in `exit_status'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/unparser-0.5.6/lib/unparser/cli.rb:64:in `run'
/opt/local/lib/ruby3.0/gems/3.0.0/gems/unparser-0.5.6/bin/unparser:10:in `<top (required)>'
Error: (string)
Good one. Since parser in the 3.x release normalizes the AST more I can probably tone back the heredoc emitter trigger. Reducing the cases for this kinds of bugs.
Still dstr emitting is the major weak point of unparser.
\o/
Thanks! Will confirm for our use case when the gem is available.
Cheers!
@akimd Released as unparser-0.5.7. https://rubygems.org/gems/unparser/versions/0.5.7
Problem solved, thanks a lot!
@akimd Actually the "re-dispatch for heredoc reminders" has to happen for every emitter that visits is descendants that can contain dstr.
So I'm 100% confident missing heredocs exist for other locations also. I've to audit the emitters, but do not find the time for the moment. I can also write a generator to replace every "variadic" child node with a DSTR to force them out.
This has to wait till I can find more time for this part of mutants infrastructure.