Step back from Token and Block types
tlienart opened this issue · comments
it's a bit annoying but it's explicitly warned against:
should probably move back to having it as a field.
Note that we don't really need to use Symbols everywhere. Since we only effectively have few tokens we could use a basic int encoding like
const T_EOS = 0
const T_NONE = -1
const T_DBB_OPEN = 1
...
the number does not matter. But then we can have bits typing like
struct Token <: AbstractSpan
id::Int
from::Int
to::Int
end
which might just be better full stop...