NQP grammar doesn't seem to work on its own
florian-pe opened this issue · comments
Hi,`
I'm not sure this is the right place to ak this question as this is not a bug per se, but it should work while it doesn't.
So overall, I'm trying to understand NQP's grammar and parser (and Raku's), and more specifically the operator precedence part.
I thought I understood the interplay between the grammar and the operator precedence parser (the method EXPR), until I asked myself: "What about parentheses ?"
This what I currently understand :
the grammar used to parse NQP programs is NQP::Grammar and is located in src/NQP/Grammar.nqp
and it also depends on HLL::Grammar from src/HLL/Grammar.nqp
The parsing starts at TOP, eventually calling EXPR, which itself basically call termish / term and infixish / infix.
My question is where are parentheses handled ?
I know that they corresponds to token circumfix:sym<( )> { '(' <.ws> <EXPR>**0..1 ')' }
I tried to look for places where this token is called, but from the very few places that it is called, there is not a single place that seems to be related to EXPR or term.
So decided to copy paste some source code into a file (I'm using rakudo-2021.07, for no particular reason).
So my test script contains :
#!/usr/bin/env nqp
# the entirety of src/HLL/Grammar.nqp
# the entirety of src/NQP/Grammar.nqp
I had to comment out a few lines from token comp_unit though. And this is what it now looks like :
token comp_unit {
:my $*IN_DECL := '';
:my $*HAS_YOU_ARE_HERE := 0;
:my $*MAIN_SUB;
# :my $*UNIT := $*W.push_lexpad($/);
# Create GLOBALish - the current GLOBAL view, created fresh
# for each compilation unit so we get separate compilation.
# :my $*GLOBALish := $*W.pkg_create_mo(self.how('knowhow'), :name('GLOBALish'));
{
$*GLOBALish.HOW.compose($*GLOBALish);
# $*W.install_lexical_symbol($*UNIT, 'GLOBALish', $*GLOBALish);
}
# This is also the starting package.
:my $*PACKAGE := $*GLOBALish;
{
$/.set_package($*PACKAGE);
# $*W.install_lexical_symbol($*UNIT, '$?PACKAGE', $*PACKAGE);
}
# Create EXPORT::DEFAULT.
:my $*EXPORT;
{
unless %*COMPILING<%?OPTIONS><setting> eq 'NULL' {
# $*EXPORT := $*W.pkg_create_mo(self.how('knowhow'), :name('EXPORT'));
# $*EXPORT.HOW.compose($*EXPORT);
# $*W.install_lexical_symbol($*UNIT, 'EXPORT', $*EXPORT);
# my $DEFAULT := $*W.pkg_create_mo(self.how('knowhow'), :name('DEFAULT'));
# $DEFAULT.HOW.compose($DEFAULT);
# ($*EXPORT.WHO)<DEFAULT> := $DEFAULT;
}
}
{
# $*W.add_initializations();
}
<.outerctx>
<statementlist>
<.set_braid_from(self)>
<.check_PACKAGE_oopsies('comp_unit')>
[ $ || <.panic: 'Confused'> ]
}
and then I added a few tests.
my $var := 5;
NQP::Grammar.parse('my $var := 5; ');
Both of those statement worked fine without generating any error.
But then, surprise !
I doesn't work when I try to parse an expression with parentheses.
# the previous $var declaration has been commented
my $var := (1 + 2); # this works fine
# but this doesn't
NQP::Grammar.parse('my $var := (1 + 2)');
I tried a few others like this one and it doesn't either.
NQP::Grammar.parse('my $var := 0+(1+2)*5; ');
The following is the complete stack trace for this particular test.
NQP::Grammar.parse('my $var := (1 + 2)');
Missing infixish operator precedence at line 2, near " 2)"
at ./nqp_parser.nqp:222 (<ephemeral file>:panic)
from ./nqp_parser.nqp:480 (<ephemeral file>:EXPR)
from <unknown>:1 (<ephemeral file>:circumfix:sym<( )>)
from gen/moar/stage2/QRegex.nqp:1704 (/home/USER/rakudo/rakudo-2022.04/install/share/nqp/lib/QRegex.moarvm:!protoregex)
from <unknown>:1 (<ephemeral file>:circumfix)
from <unknown>:1 (<ephemeral file>:term:sym<circumfix>)
from gen/moar/stage2/QRegex.nqp:1704 (/home/USER/rakudo/rakudo-2022.04/install/share/nqp/lib/QRegex.moarvm:!protoregex)
from <unknown>:1 (<ephemeral file>:term)
from <unknown>:1 (<ephemeral file>:termish)
from ./nqp_parser.nqp:412 (<ephemeral file>:EXPR)
from <unknown>:1 (<ephemeral file>:initializer)
from ./nqp_parser.nqp:1175 (<ephemeral file>:variable_declarator)
from <unknown>:1 (<ephemeral file>:declarator)
from <unknown>:1 (<ephemeral file>:scoped)
from <unknown>:1 (<ephemeral file>:scope_declarator:sym<my>)
from gen/moar/stage2/QRegex.nqp:1704 (/home/USER/rakudo/rakudo-2022.04/install/share/nqp/lib/QRegex.moarvm:!protoregex)
from <unknown>:1 (<ephemeral file>:scope_declarator)
from <unknown>:1 (<ephemeral file>:term:sym<scope_declarator>)
from gen/moar/stage2/QRegex.nqp:1704 (/home/USER/rakudo/rakudo-2022.04/install/share/nqp/lib/QRegex.moarvm:!protoregex)
from <unknown>:1 (<ephemeral file>:term)
from <unknown>:1 (<ephemeral file>:termish)
from ./nqp_parser.nqp:412 (<ephemeral file>:EXPR)
from ./nqp_parser.nqp:830 (<ephemeral file>:statement)
from <unknown>:1 (<ephemeral file>:statementlist)
from ./nqp_parser.nqp:795 (<ephemeral file>:comp_unit)
from ./nqp_parser.nqp:653 (<ephemeral file>:TOP)
from gen/moar/stage2/QRegex.nqp:2267 (/home/USER/rakudo/rakudo-2022.04/install/share/nqp/lib/QRegex.moarvm:parse)
from ./nqp_parser.nqp:1 (<ephemeral file>:<mainline>)
from gen/moar/stage2/NQPHLL.nqp:1949 (/home/USER/rakudo/rakudo-2022.04/install/share/nqp/lib/NQPHLL.moarvm:eval)
from gen/moar/stage2/NQPHLL.nqp:2154 (/home/USER/rakudo/rakudo-2022.04/install/share/nqp/lib/NQPHLL.moarvm:evalfiles)
from gen/moar/stage2/NQPHLL.nqp:2114 (/home/USER/rakudo/rakudo-2022.04/install/share/nqp/lib/NQPHLL.moarvm:command_eval)
from gen/moar/stage2/NQPHLL.nqp:2039 (/home/USER/rakudo/rakudo-2022.04/install/share/nqp/lib/NQPHLL.moarvm:command_line)
from gen/moar/stage2/NQP.nqp:4111 (/home/USER/rakudo/rakudo-2022.04/install/share/nqp/lib/nqp.moarvm:MAIN)
from gen/moar/stage2/NQP.nqp:1 (/home/USER/rakudo/rakudo-2022.04/install/share/nqp/lib/nqp.moarvm:<mainline>)
from <unknown>:1 (/home/USER/rakudo/rakudo-2022.04/install/share/nqp/lib/nqp.moarvm:<main>)
from <unknown>:1 (/home/USER/rakudo/rakudo-2022.04/install/share/nqp/lib/nqp.moarvm:<entry>)
My question is where are parentheses handled ?
EXPR uses termish to parse a term; the piece you're probably missing is that HLL::Grammar has a case of termish that calls circumfix, which is how we end up parsing parenthesized expressions, array composers, etc.
I doesn't work when I try to parse an expression with parentheses.
I don't think it's about parentheses, but rather any operator, going by the error. And that's probably in turn because the actions class is not provided, and the O rule's action method plays a role in attaching precedence info - which matches the error you receive.
it should work while it doesn't
Given the existence of BEGIN time (in both NQP and Raku) time and the requirement of a symbol table to parse Raku, NQP::Grammar can't really stand alone.
Ah yes, I don't know how is passed over token term:sym<circumfix> { <circumfix> }.
I don't think it's about parentheses, but rather any operator, going by the error. And that's probably in turn because the actions class is not provided, and the O rule's action method plays a role in attaching precedence info - which matches the error you receive.
it should work while it doesn't
Given the existence of BEGIN time (in both NQP and Raku) time and the requirement of a symbol table to parse Raku, NQP::Grammar can't really stand alone.
Yeah, it was certainly a long shot to copy/paste code, it was bound to not work for a reason or another.
I'll try to extract just the relevant part of the grammar, without the code, to be able to parse my little examples and as an excercise. But now I should have all the pieces necessary to understand and replicate the operator precedence parser. Thanks for the info!