Comment tokens for Parser
rajanmaghera opened this issue · comments
Right now, any comments are completely ignored by the parser since they are not needed. Comments are the portion of a line that begin with a # and go until the end of that line.
We would like there to be an extra Token for Comment. This token should keep track of the text inside the comment. When the tokens are getting converted to ParserNodes, we should ignore them.
There should also be some tests to verify that they are being read properly. The previous tests that exist in the lexer.rs file might not pass because they were originally ignoring the comments. Those tests should be updated to pass as well.
These could be used to mark up functions, to enforce certain argument details.