osm2pgsql-dev / osm2pgsql

OpenStreetMap data to PostgreSQL converter

Home Page:https://osm2pgsql.org

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Allow more than 32 generalization jobs

pnorman opened this issue · comments

osm2pgsql-gen currently is limited to 32 threads.

if (jobs < 1 || jobs > 32) {
log_error("The --jobs/-j parameter must be between 1 and 32.");
return 2;
}

This is limiting me to using less than 50% of my CPU.

Is there any hard limit elsewhere in the code that prevents the upper limit from being removed?

I took that limit from the --number-processes parameter. But that is a bit different than the number of jobs here. So I don't have anything against making the limit larger. I think we should have some limit, to save the user from themselves doing something ridiculous here. And the question is at which point more threads will not make this faster.

Tile based CPU bound work like this will generally parallelize until the CPU is saturated, with as many threads as there are hardware threads. The OSMF render servers have 112 threads, and they're moderate as servers go.