Implement a proper Lexer for mash scripts
raklaptudirm opened this issue · comments
@tjgurwara99 Do you think we should allow explicit semicolons in the code?
I think yes, we should. In fact, I think you should not add a semicolon at the lexer level but rather the parser level. Lexer shouldn't couple with the parser logic (for example when a statement ends). Instead it should lex it if it exists but add it in the parser since the parser would have the context in which it appears.
Since the semicolons, implicit or explicit, are a part of the token stream, I believe only the parser should manipulate them. This will decrease the overhead from the parser, and separate the concerns of the two pieces of code.