How can I use the matchLongest lexer option?
wrycode opened this issue · comments
It's a private field and I don't see a way to set it
It looks like dead code, I'll remove it. What are you trying to achieve?
My project needs to parse input with user-defined 'form' dictionaries where a form could be one or multiple characters from the input stream, and the largest should be selected first. It's for phonetics. One example is 'kw', which makes the 'q' form, where 'k' and 'w' are also individual forms when they're not found together.
Struct tags can't be changed at runtime, so I was just converting each 'form' to a token in the lexer. It's possible to generate regex like kw|(k?w) but it gets kind of messy. Now my plan is to generate a prefix tree data structure from the user's form dictionary and just write my own parser that traverses the prefix tree while scanning the input to find the longest match.
I appreciate your work on this library.