Parallel build executors

[Melroy van den Berg]

Hi,

I'm trying to build the Tor Project Homepage. This is taking way too long IMO.

I think this mainly due to the fact that Lektor is not utilizing multiple threads and executing the build in parallel.

Instead, Lektor is currently building just one file at the time, using a single thread. This doesn't make sense, I think Lektor build can really improve on this aspect.

Regards,
Melroy van den Berg

[Jeff Dairiki]

This, too, is an interesting idea.

I suspect it's doable, but not trivial. I will try to look into it (sometime after the holidays, probably.)

[Melroy van den Berg]

I also think its doable but non trivial. Just keep in mind you create a correct dependency graph (AST: abstract syntax tree) first. After having that in place, you can't go wrong.

Ps. Did you know a great build system is also built-on Python? It's called Meson. So it's definitely feasible.

[Jerome Charaoui]

Another example of a project that takes a very very long time to build: https://gitlab.torproject.org/tpo/web/blog

We've been tracking this specific issue here: Build time is out of control and witness the stupid hacks we've had to implement to work around this problem.

[Melroy van den Berg]

Yeah, it's unbearable. I really hope somebody take action and take it seriously.

[Jerome Charaoui]

For the record, the (very) long build times that are due to large numbers of content files is a major factor in the Tor Project now considering to migrate its Lektor projects to an different static site generator.

If at all possible, I'd appreciate if this issue were given extra attention in the upcoming development priorities.