Pure-python alternative to simplates

[Charly C.]

Adding support for pure-python (.py) dynamic resources, in addition to simplates, could be a solution to several issues. Here's what it could look like:

import things

do_one_time_stuff
a_global_var = None

def GET(website, response):  # dependency injection here
    if bad:
        return response.error(400, 'xxx')
    do_GET_stuff
    return locals()

def POST(request, response):
    do_POST_stuff
    response.redirect('…')

PAGES = []

PAGES.append("""text/html
<p>Lorem ipsum</p>
""")

PAGES.append("""application/json
{'lorem': 'ipsum'}
""")

# or

PAGES = """
[---] text/html
<p>Lorem ipsum</p>
[---] application/json
{'lorem': 'ipsum'}
"""

Notes:

• it's a little more verbose than a simplate, but not much
• variables aren't passed implicitly to the templates by default (AspenWeb/pando.py#215)
• pyflakes friendly (no need for AspenWeb/pando.py#30 and AspenWeb/pando.py#546)
• also more friendly to other tools, e.g. test coverage measurement
• single source of truth for supported methods (that's even better than AspenWeb/pando.py#204)
• possible to decorate functions
• for proper error reporting we can scan the file to determine the line number each template starts at
• syntax highlighting of templates is still possible, but instead of making a new syntax highlighter you need to modify the python one

@whit537 What do you think? Why did you originally go with a custom file format? Is there some issue with using pure python that I missed/forgot?

[Chad Whitacre]

Why did you originally go with a custom file format?

Before we had the .spt file extension, Aspen would attempt to process any file as a simplate. This made for a "progressively enhanced" developer experience. You could start with a foo.html file with straight HTML in it, and then add a ^L whenever you were ready and add Python.

There may be enough value in additional filetypes than simplates to warrant making that part of the system pluggable. See also: Python Server Pages.

[Charly C.]

See also: Python Server Pages.

Those are templates, like PHP files, right? They can't contain multiple pages like simplates and my proposal above can.

[Chad Whitacre]

Sure, but 0, 1, ∞. If we're supporting a second option, then the task is to make the system pluggable, to support N options.

[Charly C.]

then the task is to make the system pluggable

That was already part of AspenWeb/salon#13.

[Chad Whitacre]

Well, not necessarily. We factored out the algorithm.py library, but the part of Pando that uses the algorithm.py library isn't pluggable.

[Chad Whitacre]

@Changaco What I don't like about your proposal is the PAGES object at the bottom. Whether a list or a string, it's ugly and goes against the grain of editors and other tooling, as you suggest. What if, instead, we just admit defeat entirely on mixed-mode files?

In the past, we've toyed with the idea of splitting simplate pages out into a directory. I don't like that because it overrides what a directory means in URL-space (and what would the files inside the directory be named? foo/foo.html? foo/html?). But separate files in the original directory, on the other hand, would be a natural fit with the way non-dynamic web servers already work. Consider a publishing root with the files:

foo.html
foo.json
foo.py

Let's say that:

• if you hit http://example.com/foo.html you'd get the foo.html file,
• http://example.com/foo.json would hit foo.json, and
• http://example.com/foo would vary based on Accept.

So far, so good.

Now, let's say that if you hit http://example.com/foo.py you'd get 404. However, every request for foo.html or foo.json would consult foo.py first, according to the top half of your proposal above. Perhaps we also allow for foo.{html,json}.py for processing that is specific to those resource representations (can it import from foo.py? but its name may not be Python-safe—perhaps base is an injectable dependency?). This should work fine with wildcards; %foo.html would consult %foo.py.

Waddya think? :)

[Chad Whitacre]

I should also add that foo.html and foo.json would use renderers, they wouldn't generally themselves be plain HTML or JSON files.

[Chad Whitacre]

So on a project you'd configure your editor to highlight files according to the renderer configured for the project (Jinja2 for *.html, maybe Python for *.json). That's pretty commonly doable, yes?

[Chad Whitacre]

A security risk is that if you publish the publishing root with Apache or Nginx, you leak the source code.

If there's a foo.html but no foo.py, does foo.html still go through a renderer? Or is it served statically?

[Chad Whitacre]

Simplates are Aspen's Achilles heal. Filesystem dispatch is compelling. Multi-mode files are a liability.

[Charly C.]

Whether a list or a string, it's ugly and goes against the grain of editors and other tooling, as you suggest.

Where did I suggest that? 😕

What if, instead, we just admit defeat entirely on mixed-mode files?

Mixed-mode files are only a problem because of stupid tools and lack of standardization (for example I'm not aware of a PEP specifying how to tag a python string as containing a specific media type or language, maybe we should write one). I'm not sure fleeing the problem is a good solution: are you also going to stop using SQL in gratipay's python code because it's mixing two languages in one file?

Waddya think? :)

Routing /foo.html to foo.py instead of foo.html would require modifying the dispatcher, breaking the "most specific first" rule, and that worries me. Using foo.py.html instead could be a solution, albeit a slightly weird one.

Overall your proposal seems pretty compatible with mine. Either:

• PAGES could be optional, though it would create an ambiguity when both PAGES and the separate files exist; or
• PAGES could have support for template paths (e.g. text/html from foo.py.html), though that would be less convenient (but more explicit and flexible)

Multi-mode files are a liability.

They're also an asset: less time spent switching between files.

[Chad Whitacre]

Where did I suggest that?

Up at "you need to modify the python one," and also in your subsequent paragraph about "stupid tools and lack of standardization." ;-)

Routing /foo.html to foo.py instead of foo.html

Instead? I see this as routing to foo.html through foo.py. foo.py is never directly addressable through the web.

Overnight I've come to think that we should include the renderer in the filename—foo.html.jinja2. This aids tooling, resolves the static/dynamic question, and is parallel to how we handle *.spt files already: foo.html.{spt,jinja2} is more specific than foo.html. Actually, we could implement this to be flexible: files are "renderable" based on file extensions registered with Aspen, and any renderable file runs through foo.py (if there is one). That way, if you wanted to register .html instead of .jinja2, you could.

less time spent switching between files.

One either switches between files, or between locations in a single file. Having the files all right next to each other in the same directory at least makes opening them all at once easier. The question is whether the benefit of having them in the same file outweighs the cost of fighting against "stupid tools and lack of standardization." There's no shortage of work in the world. After nine years, I for one am ready to work on other things besides building an ecosystem around multi-mode files. ;-)

[Charly C.]

Up at "you need to modify the python one,"

That's not really more difficult than creating one for simplates. I was comparing simplates to python files containing a PAGES variable, your reply was comparing the latter to external template files. We weren't on the same page. :D

foo.html.{spt,jinja2} is more specific than foo.html

The current dispatcher will pick the latter first. This property is used in Gratipay/Liberapay to serve compiled assets.

Another dispatch dilemma we haven't considered yet is what to do when there's both a .spt and a .py file. I guess the solution would be to raise an exception or warning.

The question is whether the benefit of having them in the same file outweighs the cost of fighting against "stupid tools and lack of standardization."

As I said, even if we gave up simplates and the PAGES idea we'd still have the mixed-mode "stupid tools and lack of standardization" problem in other contexts such as SQL queries in python code.

After nine years, I for one am ready to work on other things besides building an ecosystem around multi-mode files. ;-)

So you want to abandon simplates?

[Charly C.]

Regarding mixed-mode files, I also want to point out that they are in fact very common, for example here's some HTML with CSS inside:

<h1>Lorem ipsum</h1>
<style>h1 {font-size: 20px;}</style>

As you can see GitHub highlights the CSS by default, and so does my text editor, because it's common and standard to embed CSS in HTML.

[Charly C.]

(GitHub even respects the type attribute:

<script type="application/javascript">this.is("javascript");</script>
<script type="text/foobar">this is not javascript</script>

My editor doesn't. 😞 )

[Chad Whitacre]

here's some HTML with CSS inside
SQL queries in python code

Fair enough. It seems that there is some ecosystem support for mixed-mode files for common cases, and little support for uncommon cases.

So you want to abandon simplates?

Quite possibly! I I've started #30 to explore this further and see if it's at all doable before 1.0. 😁

[Charly C.]

I think I figured out how we can modify the dispatcher to allow implementing your multiple-files idea without breaking other stuff. Bonus: with this change a foo.html[.spt] file will be accessible at /foo, while currently it's only accessible at /foo.html.

[Chad Whitacre]

Cool. I don't think we should try to make any more progress on this before 1.0, however (that's why I cut off #30 early).