Syntax error messages lack specificity available in modern Python

[ascholerChemeketa]

Python 3.10 made major improvements to the error messaging around syntax errors. The difference between what modern versions of python provide and skulpt is pretty dramatic. (Examples below.)

One significant use of skulpt is to enable browser based "learn programming with python" books/tools. For this use case, better syntax errors would be a tremendous boon.

How complex would it be to add more detailed syntax help? Even adding a subset of basic tips for core programming constructs would represent a significant improvement for beginning programmers using a skulpt based system.

Sample errors:

Code

x = 95
if x == 100
    print("Perfect score")

Python 3.10

    if x == 100
               ^
SyntaxError: expected ':'

Skulpt

SyntaxError: bad input on line 2

---

Code

x = 0
while x < 10:
print(x)
x = x + 1

Python 3.10

    print(x)
    ^
IndentationError: expected an indented block after 'while' statement on line 3

Skulpt

SyntaxError: bad input on line 3

[stu]

There are two phases for these sorts of error messages:

1. SyntaxErrors from tokenizing
2. SyntaxErrors from parsing

We have done some work on a new Parser, which will hopefully get to production at some point.
The new Parser should match CPython in terms of SyntaxErrors 😄

But the Tokenizing phase is tricky and would require a rewrite of our tokenizer.
Skulpt uses the python lib version of tokenize, which is good enough, but it's not what CPython actually uses to tokenize a python program.
To get parity with CPython I think we'd need to rewrite the tokenizer 😬