unexpected token error

[Jesse Wise]

Marko Version: 5.20.9 and newer

Details

a production build command now fails with this error:
[marko-vite:pre] src\components\meta.marko(127,107): Unexpected token
125 | meta [
126 |

127 | content="https://image.domain.com/" +
| ^
128 | encodeURIComponent(featuredImgMarko).replace("'", "%27")
129 | ]

Expected Behavior

this error isn't thrown in 5.20.8 or older compilers. guessing the new htmljs-parser codebase may have introduced this issue, but not sure.

Actual Behavior

the above error is thrown

Possible Fix

not sure, maybe it no longer allows string concatenation like in the example.

Your Environment

• Environment name and version: Node 16.14.0
• Operating System and version (desktop or mobile): Mac and Windows

Steps to Reproduce

1. npm run build from within a vite-express based project

Stack Trace

closest thing to stack trace is in the top level description

[Jesse Wise]

here's an example file that produces the error when using 5.20.9 and later, but not 5.20.8:

html
  head
    meta [
      
      content="https://test.image.com/" +
      encodeURIComponent("image.jpg").replace("'", "%27")
    ]
  body

[Dylan Piercey]

@jwise7 this is a tricky one. There was some issues in the parser around how it handles expression continuations across new lines, lets take your example and show you the issues:

• parses fine in older versions, but not now (as you said)

    meta [
      content="https://test.image.com/" +
      encodeURIComponent("image.jpg").replace("'", "%27")
    ]

• errors in all versions (just removing the space before the plus)

    meta [
      content="https://test.image.com/"+
      encodeURIComponent("image.jpg").replace("'", "%27")
    ]

• works in previous versions (remove leading space, add space after)

    meta [
      content="https://test.image.com/"+ 
      encodeURIComponent("image.jpg").replace("'", "%27")
    ]

In general the parser wasn't really designed to do these expression continuations across lines. However if the expression with newlines is enclosed in brackets it always works. eg:

    meta [
      content=("https://test.image.com/"+
      encodeURIComponent("image.jpg").replace("'", "%27"))
    ]

or

    meta [
      content=(
        "https://test.image.com/" +
        encodeURIComponent("image.jpg").replace("'", "%27")
      )
    ]

Although this change does break code like this, we decided it was more in the bug territory.
We were hoping that in practice it wouldn't come up much and that the work around above was straightforward enough.

It may be something we reconsider, but this is where we're at so far.

[Jesse Wise]

the real issue may be with marko-prettier forcing one liners into multiple lines without the parens. and, removing the parens when i tried to add them.

[Jesse Wise]

our marko-prettier approved work around looks like this: (it kept deleting the parens when I tried that method)

meta [
  
  content=`https://test.image.com/${encodeURIComponent(
     "image.jpg"
  ).replace("'", "%27")}`
]