sapk / tegola-osm

Various scripts for importing and running a mirror of OSM with tegola

Tegola OSM

This repo houses instructions and configuration files to aid with standing up an OpenStreetMap export and Natural Earth dataset into a PostGIS enabled database that uses tegola for creating and serving vector tiles.

Repo config files

• imposm3.json - an imposm3 mapping file for the OSM PBF file.
• tegola.toml - a tegola configuration file for the OSM import produced by imposm3.

Dependencies

• Postgres server with PostGIS enabled.
• imposm3 (download - linux only)
• tegola (download)
• gdal - required for Natural Earth import

Download the OSM planet database in PBF format

curl -O https://planet.openstreetmap.org/pbf/planet-latest.osm.pbf

Import the OSM export into PostGIS using imposm3

./imposm3 import -connection postgis://username:password@host/database-name -mapping imposm3.json -read /path/to/osm/planet-latest.osm.pbf -write
./imposm3 import -connection postgis://username:password@host/database-name -mapping imposm3.json -deployproduction

Import the OSM Land and Natural Earth dataset (requires gdal, Natural Earth can be skipped if you're only interested in OSM)

Option 1: Embed Credentials

Update the database credentials inside of natural_earth.sh and osm_land.sh, then run each file: ./natural_earth.sh && ./osm_land.sh. This will download the natural earth and osm land datasets and insert it into PostGIS under a database named natural_earth and osm respectively.

Option 2: Create a dbcredentials.sh file

Create a dbcredentials.sh file which will be shared with the osm_land script. This option is ideal for when the natural_earth and osm databases will reside on the same database server, and will use the same credentials. Ensure that the following variables are defined in your file:

DB_HOST="mydbhost"
DB_PORT="myport"
DB_USER="myuser"
DB_PW="mypassword"

Once you have configured the dbcredentials.sh file, run the scripts as above: ./natural_earth.sh && ./osm_land.sh

Option 3:

Create separate configuration files in the same pattern as the above dbcredentials.sh file and pass the path to the config file using the -c option. This is ideal if you have two different servers for the databases. Ensure the file you create follows this format:

DB_NAME="mydb"
DB_HOST="mydbhost"
DB_PORT="myport"
DB_USER="myuser"
DB_PW="mypassword"

Once you have configured the files, run the scripts with the -c flag and provide the path to the credentials file, ie: ./natural_earth.sh -c natural_earth_creds.sh && ./osm_land.sh -c osm_creds.sh

Usage:

Both scripts support a -v flag for debugging. natural_earth.sh also supports a -d flag, which will drop the existing natural earth database prior to import if set. Since the osm_land.sh imports into a database shared with other data, it lacks this functionality. Instead, only the relevent tables are dropped.

Install SQL helper functions

Execute postgis_helpers.sql against your OSM database. Currently this contains a single utility function for converting building heights from strings to numbers which is important if you want to extrude buildings for the 3d effect.

psql -U tegola -d database-name -a -f postgis_helpers.sql

Setup SQL indexes

Execute postgis_index.sql against your OSM database.

psql -U tegola -d database-name -a -f postgis_index.sql

Launch tegola

./tegola -config=tegola.toml

Open your browser to localhost and the port you configured tegola to run on (i.e. localhost:8080) to see the built in viewer.

Data Layers

To view these data layers in a map and query the features for a better understanding of each data layer, use the Tegola-OSM Inspector. The data layers described here are in the "Tegola-OSM" database as laid out in the tegola.toml (i.e., not the Natural Earth database that is specified in tegola-natural-earth.toml).

source	Description
ne	Natural Earth data, version 4
osm	OpenStreetMap data, current
osm land	OpenStreetMap-derived land polygons from openstreetmapdata.com, currentness depends on last pull

Note: All layers also have the data fields: layer id and geometry. An empty where column means that all features are retained.

populated_places

points

zoom	source	table/layer	data fields	where
0-2	ne	ne_110m_populated_places	scalerank, labelrank, name, min_zoom, featurecla, rank_max
3-4	ne	ne_50m_populated_places	scalerank, labelrank, name, min_zoom, featurecla, rank_max
5-20	ne	ne_10m_populated_places	scalerank, labelrank, name, min_zoom, featurecla, rank_max

country_lines

zoom	source	table/layer	data fields	where
0-2	ne	ne_110m_admin_0_boundary_lines_land	featurecla, name, min_zoom
3-4	ne	ne_50m_admin_0_boundary_lines_land	featurecla, name, min_zoom
5-10	ne	ne_10m_admin_0_boundary_lines_land	featurecla, name, min_zoom

country_lines_disputed

lines

zoom	source	table/layer	data fields	where
3-4	ne	ne_50m_ne_50m_admin_0_boundary_lines_disputed_areas	featurecla, name, min_zoom
5-10	ne	ne_10m_ne_50m_admin_0_boundary_lines_disputed_areas	featurecla, name, min_zoom

country_label_points

zoom	source	table/layer	data fields	where
3-20	ne	ne_10m_admin_0_label_points	sr_subunit, scalerank

country_polygons

zoom	source	table/layer	data fields	where
0-2	ne	ne_110m_admin_0_countries	featurecla, name, name_long, abbrev, adm0_a3, min_zoom, min_label, max_label
3-4	ne	ne_50m_admin_0_countries	featurecla, name, name_long, abbrev, adm0_a3, min_zoom, min_label, max_label
5-10	ne	ne_10m_admin_0_countries	featurecla, name, name_long, abbrev, adm0_a3, min_zoom, min_label, max_label

state_lines

zoom	source	table/layer	data fields	where
0-2	ne	ne_110m_admin_1_states_provinces_lines	featurecla, name, adm0_name, min_zoom
3-4	ne	ne_50m_admin_1_states_provinces_lines	featurecla, name, adm0_name, min_zoom
5-10	ne	ne_10m_admin_1_states_provinces_lines	featurecla, name, adm0_name, min_zoom

land

polygons

zoom	source	table/layer	data fields	where
0-2	ne	ne_110m_land	featurecla, min_zoom
3-4	ne	ne_50m_land	featurecla, min_zoom
5-7	ne	ne_10m_land	featurecla, min_zoom
8-20	osm land	land_polygons

admin_lines

zoom	source	table/layer	data fields	where
8-12	osm	admin_boundaries_8-12	admin_level, name, type	admin_level IN (1,2,3,4,5,6,7,8)
13-20	osm	admin_boundaries_13-20	admin_level, name, type	admin_level IN (1,2,3,4,5,6,7,8,9,10)

state_label_points

zoom	source	table/layer	data fields	where
3-20	ne	ne_10m_admin_1_label_points	name, scalerank

landuse_areas

Nature reserves, military land, forest, leisure, wood, etc. polygons

zoom	source	table/layer	data fields	where
3-5	osm	landuse_areas_gen0	name, class, type, area	type IN ('forest','wood','nature reserve', 'nature_reserve', 'military') AND area > 1000000000
6-9	osm	landuse_areas_gen0_6	name, class, type, area	type IN ('forest','wood','nature reserve', 'nature_reserve', 'military') AND area > 100000000
10-12	osm	landuse_areas_gen1	name, class, type, area
13-20	osm	landuse_areas	name, class, type, area

water_areas

polygons

zoom	source	table/layer	data fields	where
3-5	osm	water_areas_gen0	name, class, type, area	type IN ('water', 'pond', 'basin', 'canal', 'mill_pond', 'riverbank') AND area > 1000000000
6-9	osm	water_areas_gen0_6	name, class, type, area	type IN ('water', 'pond', 'basin', 'canal', 'mill_pond', 'riverbank') AND area > 100000000
10-12	osm	water_areas_gen1	name, class, type, area	type IN ('water', 'pond', 'basin', 'canal', 'mill_pond', 'riverbank') AND area > 1000
13-20	osm	water_areas	name, class, type, area	type IN ('water', 'pond', 'basin', 'canal', 'mill_pond', 'riverbank', 'dock')

water_lines

zoom	source	table/layer	data fields	where
8-12	osm	water_lines_gen0	name, type	type IN ('river', 'canal')
13-14	osm	water_lines_gen1	name, type	type IN ('river', 'canal', 'stream', 'ditch', 'drain', 'dam')
15-20	osm	water_lines	name, type	type IN ('river', 'canal', 'stream', 'ditch', 'drain', 'dam')

transport_lines

Roads, airport runways, ferry routes, paths, etc.

zoom	source	table/layer	data fields	where
3-4	ne	ne_10m_roads_3	name, min_zoom, min_label, type, label	min_zoom < 5 AND type <> 'Ferry Route'
5-6	ne	ne_10m_roads_5	name, min_zoom, min_label, type, label	min_zoom <= 7 AND type <> 'Ferry Route'
7-8	osm	transport_lines_gen0	type, tunnel, bridge, ref	type IN ('motorway','trunk','motorway_link','trunk_link','primary') AND tunnel = 0 AND bridge = 0
9-10	osm	transport_lines_gen1	ref, class, type	type IN ('motorway', 'trunk', 'primary', 'primary_link', 'secondary', 'motorway_link', 'trunk_link')
11-12	osm	transport_lines_11-12	name, ref, class, type, tunnel, bridge, access, service	type IN ('motorway', 'motorway_link', 'trunk', 'trunk_link', 'primary', 'primary_link', 'secondary', 'secondary_link', 'tertiary', 'tertiary_link', 'rail', 'taxiway', 'runway', 'apron')
13	osm	transport_lines_13	name, ref, class, type, tunnel, bridge, access, service	type IN ('motorway', 'motorway_link', 'trunk', 'trunk_link', 'primary', 'primary_link', 'secondary', 'secondary_link', 'tertiary', 'tertiary_link', 'rail', 'residential', 'taxiway', 'runway', 'apron')
14-20	osm	transport_lines_14-20	name, ref, class, type, tunnel, bridge, access, service

transport_areas

Airports, etc. polygons

zoom	source	table/layer	data fields	where
12-20	osm	transport_areas	name, class, type

transport_points

Airports, helipads, etc.

zoom	source	table/layer	data fields	where
14-20	osm	transport_points	name, class, type

amenity_areas

Fire stations, banks, embassies, government, police stations, schools, universities, etc. polygons

zoom	source	table/layer	data fields	where
14-20	osm	amenity_areas	name, type

amenity_points

Fire stations, banks, embassies, government, police stations, schools, universities, etc.

zoom	source	table/layer	data fields	where
14-20	osm	amenity_points	name, type

other_points

Man made, historic, military, barriers, power towers, etc.

zoom	source	table/layer	data fields	where
14-20	osm	other_points	name, class, type

other_lines

Man made, historic, military, barriers, power lines, etc.

zoom	source	table/layer	data fields	where
14-20	osm	other_lines	name, class, type

other_areas

polygons Man made, historic, military, power, barriers, piers, etc.

zoom	source	table/layer	data fields	where
6-8	osm	other_areas_filter	name, class, type	area > 1000000
9-20	osm	other_areas	name, class, type

buildings

polygons

zoom	source	table/layer	data fields	where
14-20	osm	buildings	name, height, type

How long does it take to import the entire planet?

If you run this import, please send in a PR to report your import machine specs and how long it takes.

@peldhose: 11.30 hours on a Google cloud server with 8 vCPU, 30GB RAM and 1TB storage (400GB used)