Need a strategy for string literal tokens
GavinHigham opened this issue · comments
The current lexer implementation is a list of strings, each a token (Lua does not have enums, and strings are interned for fast comparison).
So the parser can distinguish between names and strings, I need to mark string literals somehow. Currently, I just leave the quotes / brackets on them - but that will complicate the parser logic. I should probably normalize them to a single quoted string type, and keep a lookup table to the original string token (for transpiling).
Fixed by making each token a table with a "type" and "token" field.