Tokens inside of macro call and attr

[David Tolnay]

TokenTree is currently defined with the rule '(' ')' | '{' '}' | '[' ']'. I would expect the rule to allow tokens within those delimiters, as well as other kinds of tokens like identifiers and literals and punctuation.

ungrammar/rust.ungram

Lines 60 to 66 in 07d5806

 MacroCall = 
   Attr* Path '!' Name? TokenTree ';'? 
  
 TokenTree = 
   '(' ')' 
 | '{' '}' 
 | '[' ']' 

ungrammar/rust.ungram

Lines 286 to 287 in 07d5806

 Attr = 
   '#' '!'? '[' Path ('=' Literal | TokenTree)? ']' 

[Alex Kladov]

Yeah, this is somewhat of a placeholder node at the moment. We can list every token here, but that doesn’t seems useful for the purpose of genearating a syntax tree, and probably will be confusing with respect to contextual tokens & jointness.

Definitelly needs a comment in the source thoug.

[Michael Bradshaw]

Instead of trying to list every possible token, could just a generic 'token'* be used? That is:

 TokenTree = 
   '(' 'token'* ')' 
 | '{' 'token'* '}' 
 | '[' 'token'* ']' 

[Lukas Wirth]

Closing as the grammar file is no longer maintained