Lexing errors in nested OBJECT blocks masked by a StopIteration exception

[Kevin Loney]

Describe the bug
Lexing errors in nested object blocks are being masked by a StopIteration exception.

To Reproduce

example.py

import pvl
import sys

try:
	with open(sys.argv[1], 'rb') as f:
		module = pvl.load(f)
except pvl.exceptions.LexerError as ex:
	print(f'{ex.msg} - {ex.lineno}:{ex.colno}')

LEVEL1.LBL

PDS_VERSION_ID = PDS3

OBJECT = LEVEL_1
  VALUE = 123
END_OBJECT = LEVEL1

END

LEVEL2.LBL

PDS_VERSION_ID = PDS3

OBJECT = LEVEL_1
  OBJECT = LEVEL_2
    VALUE = 123
  END_OBJECT = LEVEL2
END_OBJECT = LEVEL_1

END

Expected behavior

$> python example.py LEVEL1.LBL
Expecting a Block-Name after "END_OBJECT =" that matches "LEVEL_1", but found: "LEVEL1" - 5:14

$> python example.py LEVEL2.LBL
Expecting a Block-Name after "END_OBJECT =" that matches "LEVEL_2", but found: "LEVEL2" - 6:16

Error logs or terminal captures

LEVEL1.LBL

$> python example.py LEVEL1.LBL
Expecting a Block-Name after "END_OBJECT =" that matches "LEVEL_1", but found: "LEVEL1" - 5:14

LEVEL2.LBL

$> python example.py LEVEL2.LBL
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pvl/lexer.py", line 428, in lexer
    yield None
ValueError: Expecting a Block-Name after "END_OBJECT =" that matches "LEVEL_2", but found: "LEVEL2"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pvl/parser.py", line 328, in parse_aggregation_block
    agg.append(*self.parse_aggregation_block(tokens))
  File "/usr/local/lib/python3.9/site-packages/pvl/parser.py", line 342, in parse_aggregation_block
    self.parse_end_aggregation(begin, block_name, tokens)
  File "/usr/local/lib/python3.9/site-packages/pvl/parser.py", line 458, in parse_end_aggregation
    tokens.throw(
  File "/usr/local/lib/python3.9/site-packages/pvl/lexer.py", line 435, in lexer
    raise LexerError(err, s, i, lexeme)
pvl.exceptions.LexerError: (LexerError(...), 'Expecting a Block-Name after "END_OBJECT =" that matches "LEVEL_2", but found: "LEVEL2": line 6 column 16 (char 95) near " END_OBJECT ="')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pvl/parser.py", line 511, in parse_assignment_statement
    t = next(tokens)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pvl/parser.py", line 331, in parse_aggregation_block
    agg.append(*self.parse_assignment_statement(tokens))
  File "/usr/local/lib/python3.9/site-packages/pvl/parser.py", line 898, in parse_assignment_statement
    return super().parse_assignment_statement(tokens)
  File "/usr/local/lib/python3.9/site-packages/pvl/parser.py", line 520, in parse_assignment_statement
    raise ValueError(
ValueError: Ran out of tokens before starting to parse an Assignment-Statement.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "example.py", line 11, in <module>
    module = pvl.load(f)
  File "/usr/local/lib/python3.9/site-packages/pvl/__init__.py", line 61, in load
    return loads(
  File "/usr/local/lib/python3.9/site-packages/pvl/__init__.py", line 201, in loads
    return parser.parse(s)
  File "/usr/local/lib/python3.9/site-packages/pvl/parser.py", line 831, in parse
    return super().parse(nodash)
  File "/usr/local/lib/python3.9/site-packages/pvl/parser.py", line 207, in parse
    module = self.parse_module(tokens)
  File "/usr/local/lib/python3.9/site-packages/pvl/parser.py", line 256, in parse_module
    parsed = p(tokens)
  File "/usr/local/lib/python3.9/site-packages/pvl/parser.py", line 342, in parse_aggregation_block
    self.parse_end_aggregation(begin, block_name, tokens)
  File "/usr/local/lib/python3.9/site-packages/pvl/parser.py", line 430, in parse_end_aggregation
    end_agg = next(tokens)
StopIteration

Your Environment (please complete the following information):

• OS: macOS and Linux
• Environment information: python 3.9.8 and 3.9.9
• pvl Version: 1.2.1 and 1.3.0

Additional context
None

[Ross Beyer]

Thanks, @easyas314159, yes, this is not ideal. Most likely a result of me trying to be too clever in the parsing/exception logic in parser.py. The recursive nature of nested blocks doesn't help, either.

It may take me a few days to get to this.

[Kevin Loney]

@rbeyer If it would help I'll have some time today or tomorrow and can write unit tests for this case

[Ross Beyer]

I'm not gonna say no when someone volunteers to write unit tests.

It could help, but ultimately we need to untangle the nested logic of exceptions and figure out why the parser keeps going (and then runs out of tokens with StopIteration) even after it gets that LexerError that should stop everything.

[Ross Beyer]

Ah, I figured it out. I added a simplified version of your level1 example that performed as expected and your level2 to the test_parser.py unit test. Turns out I need to remember that our LexerErrors are derived from ValueErrors, so if we're in a situation where both specific LexerErrors can get emitted or regular ValueErrors, and we want to do something different depending on which happens, I also need to catch the LexerErrors.

Looks like there are some shenanigans going on with the GitHub backends, so the tests aren't running, but I'm hoping that'll clear up on its own.

[Ross Beyer]

This fix is now merged into main, and will show up in the next release.