pallets / jinja

It's not a common use case, but I would find it very useful to be able to tweak how the Parser works. For my use case, I'd like to tweak the keywords the Parser recognises (for, if, block...) in order to restrict them as necessary (but not extend them, as that would involve changing the actual behaviour of the Parser, which I'm not aiming to tackle here).

Currently the Parser recognises the keywords from a module-level constant, which is not easily editable just for 1 Parser class.

jinja/src/jinja2/parser.py

Lines 18 to 33 in 953acd6

    
           _statement_keywords = frozenset( 
        
               [ 
        
                   "for", 
        
                   "if", 
        
                   "block", 
        
                   "extends", 
        
                   "print", 
        
                   "macro", 
        
                   "include", 
        
                   "from", 
        
                   "import", 
        
                   "set", 
        
                   "with", 
        
                   "autoescape", 
        
               ] 
        
           )

Besides, even if it were possible to specifically change a Parser, say by subclassing it and dynamically overwriting some of the methods, we would need to further subclass an Environment and override some of its methods, as the Parser is hardcoded in them.

jinja/src/jinja2/environment.py

Lines 615 to 619 in 953acd6

    
           def _parse( 
        
               self, source: str, name: t.Optional[str], filename: t.Optional[str] 
        
           ) -> nodes.Template: 
        
               """Internal parsing function used by `parse` and `compile`.""" 
        
               return Parser(self, source, name, filename).parse()

I would find much easier to implement this behaviour if the Parser had a class-level variable that could be swapped in an inheriting class, and the same with a class-level variable for the Parser in the Environment class. This is a pattern that's already followed for other internals, such as the CodeGenerator class via the code_generator_class class-variable.

jinja/src/jinja2/environment.py

Line 284 in 953acd6

code_generator_class: t.Type["CodeGenerator"] = CodeGenerator

See #1194. I plan to rewrite the parser and make it more extensible, so adding this right now only for the entire API to change doesn't seem ideal.

In your case, you can probably use an extension: https://jinja.palletsprojects.com/en/3.1.x/extensions/#writing-extensions

Or overriding Environment.parse is fine, it's the only part of the parser API that's publicly documented.

Thanks for the suggestions @davidism!

Regarding the extensions, I don't think I can make use of that, because they seem to be only taken into account if the token does not match any of the predefined values, but what I need to do is to specifically override some of the default _statement_keywords.

jinja/src/jinja2/parser.py

Lines 173 to 182 in 953acd6

    
           if token.value in _statement_keywords: 
        
               f = getattr(self, f"parse_{self.stream.current.value}") 
        
               return f()  # type: ignore 
        
           if token.value == "call": 
        
               return self.parse_call_block() 
        
           if token.value == "filter": 
        
               return self.parse_filter_block() 
        
           ext = self.extensions.get(token.value) 
        
           if ext is not None: 
        
               return ext(self)

About Environment.parse, that's what I'm currently doing, and it's not the end of the world indeed, it just felt cleaner to have the class variable to configure it. I would still need to change the behaviour of the Parser itself, though.

Finally, #1194 looks exciting! I see that's still a work in progress and there seems to be quite some road ahead. I am guessing it will include some breaking changes in order to establish the new API when the time comes, so I was wondering if the changes I proposed could bridge the gap in the meantime? I could really use them instead of dynamically faffing with the available methods in the Parser, or overriding the corresponding methods and re-writing them in a child class. I managed to do it, so it's not a massive blocker, I was just feeling that something like this approach would make things simpler.

Anyway, I appreciate it might still not be the approach you want to go for in any case and would rather keep it as is 🙂 Thanks for taking a look!

it just felt cleaner to have the class variable to configure it

Subclassing and overriding methods is an accepted and standard way to extend things in both Python and Jinja.

	_statement_keywords = frozenset(
	[
	"for",
	"if",
	"block",
	"extends",
	"print",
	"macro",
	"include",
	"from",
	"import",
	"set",
	"with",
	"autoescape",
	]
	)

	def _parse(
	self, source: str, name: t.Optional[str], filename: t.Optional[str]
	) -> nodes.Template:
	"""Internal parsing function used by `parse` and `compile`."""
	return Parser(self, source, name, filename).parse()

	if token.value in _statement_keywords:
	f = getattr(self, f"parse_{self.stream.current.value}")
	return f() # type: ignore
	if token.value == "call":
	return self.parse_call_block()
	if token.value == "filter":
	return self.parse_filter_block()
	ext = self.extensions.get(token.value)
	if ext is not None:
	return ext(self)

Configurable Parser