Parser fails to parse a heredoc that includes the delimiter in the body

[Jenia Varavva]

For example:

attr = <<DOC
  This is the the heredoc body containing the delimiter: DOC;
DOC

HCL doc says the heredoc final delimiter has to be:

The delimiter word you chose, alone on its own line (with indentation allowed for indented heredocs)

But the current grammar:

heredoc_template : /<<(?P<heredoc>[a-zA-Z][a-zA-Z0-9._-]+)\n(?:.|\n)*?(?P=heredoc)/
heredoc_template_trim : /<<-(?P<heredoc_trim>[a-zA-Z][a-zA-Z0-9._-]+)\n(?:.|\n)*?(?P=heredoc_trim)/

considers the DOC inside the body of the string to be the heredoc delimiter, despite not being on its own line.

A naive update to the grammar would be to add \n\s* in front of the final (?P=heredoc_trim). This however breaks the grammar for the empty heredoc:

foo = <<BAR
BAR

where a single \n between the two lines is consumed by the \n after the leading delimiter: <<(?P<heredoc>[a-zA-Z][a-zA-Z0-9._-]+)\n.

Since lark doesn't support regexp "lookbehinds", a more elaborate solution would be to rewrite the grammar and transformer to:

1. The parser to not consume the \n after the leading delimiter; use a lookahead instead:
   • <<(?P<heredoc>[a-zA-Z][a-zA-Z0-9._-]+)\n -->
   • <<(?P<heredoc>[a-zA-Z][a-zA-Z0-9._-]+)(?=\n)
2. This would leave an extra \n for the final delimiter in the empty string case. With the current transformer, this would also leave the leading \n in the string body (and consume the final one which is currently being trimmed by the transformer), so the transformer will also have to change to remove one leading \n from the string.

This sounds like it could be more pretty, so maybe there's a better way to achieve this.