CRLF lines not supported

[lbyonder]

Bug Report

Overview

Windows style line endings give a syntax error.

Expected

Documentation should explain what line endings are supported. Does webassembly itself support CRLF?

[Arthur Buldauskas]

Thank you for reporting this. Seems like a straightforward bug in the tokenizer. We don't match Window line endings only \n so it's invalid syntax. WebAssembly itself isn't opinionated about that type of thing because it's a binary format.

walt/packages/walt-syntax/src/tokens.js

Lines 76 to 102 in 7ca5eb8

 export const tokens = { 
   whitespace: /[ \t]+/, 
   comment: [ 
     { match: /\/\/.*?$/ }, 
     { match: /\/\*[^]*?\*\//, lineBreaks: true }, 
   ], 
   number: [ 
     { match: /0[xX][0-9a-fA-F]+/ }, 
     { match: /0[oO][0-9]+/ }, 
     { match: /0[bB][01]+/ }, 
     { match: /(?:[0-9]+(?:\.[0-9]+)?e-?[0-9]+)/ }, 
     { match: /[0-9]+\.[0-9]+|[0-9]+/ }, 
   ], 
   char: [ 
     { match: /'(?:\\['\\bfnrtv0]|[^'\\\n])'/, value: x => x.slice(1, -1) }, 
   ], 
   string: [ 
     { match: /"(?:\\["\\rn]|[^"\\\n])*?"/, value: x => x.slice(1, -1) }, 
     { match: /'(?:\\['\\bfnrtv0]|[^'\\\n])*?'/, value: x => x.slice(1, -1) }, 
     { match: /`(?:\\['\\bfnrtv0]|[^'\\])*?`/, value: x => x.slice(1, -1) }, 
   ], 
   identifier: { 
     match: /[A-Za-z_$][A-Za-z0-9_$]*/, 
     keywords: { keyword, type }, 
   }, 
   punctuator, 
   newline: { match: /\n/, lineBreaks: true }, 

Should be straightforward to fix with a better regex for newline.

[Arthur Buldauskas]

Fixed by #154

I'll publish a patched version to npm later today.