Imposm 3
Imposm is an importer for OpenStreetMap data.
Debian installer
apt-get install golang git mercurial libleveldb-dev libgeos-dev libgeos++-dev osmosis python-nose python-psycopg2 python-shapely postgis
mkdir src/github.com/omniscale/ && cd src/github.com/omniscale/ && git clone https://github.com/jmikedupont/imposm3.git
export GOPATH=`pwd`
go get github.com/tools/godep
make
Details
It reads PBF files and imports the data into PostgreSQL/PostGIS. It can also update the DB from diff files.
It is designed to create databases that are optimized for rendering (i.e. generating tiles or for WMS services).
Imposm 3 is written in Go and it is a complete rewrite of the previous Python implementation. Configurations/mappings and cache files are not compatible with Imposm 2, but they share a similar architecture.
It is released as open source under the Apache License 2.0.
The development of Imposm 3 was sponsored by Omniscale and development will continue as resources permit. Please get in touch if you need commercial support or if you need specific features.
Features
- High-performance
- Diff support
- Custom database schemas
- Generalized geometries
In detail
-
High performance: Parallel from the ground up. It distributes parsing and processing to all available CPU cores.
-
Custom database schemas: Creates tables for different data types. This allows easier styling and better performance for rendering in WMS or tile services.
-
Unify values: For example, the boolean values
1
,on
,true
andyes
all becomeTRUE
. -
Filter by tags and values: Only import data you are going to render/use.
-
Efficient nodes cache: It is necessary to store all nodes to build ways and relations. Imposm uses a file-based key-value database to cache this data.
-
Generalized tables: Automatically creates tables with lower spatial resolutions, perfect for rendering large road networks in low resolutions.
-
Limit to polygons: Limit imported geometries to polygons from Shapefiles or GeoJSON, for city/state/country imports.
-
Easy deployment: Single binary with only runtime dependencies to common libs (GEOS, SQLite and LevelDB)
-
Support for table namespace (PostgreSQL schema)
Performance
Imposm 3 is much faster than Imposm 2 and osm2pgsql:
- Makes full use of all available CPU cores
- Bulk inserts into PostgreSQL with
COPY FROM
- Efficient intermediate cache for reduced IO load during ways and relations building
Some import times from a Hetzner EX 4S server (Intel i7-2600 CPU @ 3.40GHz, 32GB RAM and 2TB software RAID1 (2x2TB 7200rpm SATA disks)) for imports of a 20.5GB planet PBF (2013-06-14) with generalized tables:
- 6:30h in normal-mode
- 13h in diff-mode
osm2pgsql required between 2-8 days in a similar benchmark (slide 7) with a smaller planet PBF file (~15GB).
Benchmarks with SSD are TBD.
Import of Europe 11GB PBF with generalized tables:
- 2:20h in normal-mode
Current status
Imposm 3 is in alpha stadium and there is no official release yet. The import itself is working however and it is already used for production databases.
Missing
Compared to Imposm 2:
- Documentation
- Support for other projections than EPSG:3857
- Import of XML files
Other missing features:
- Automatic download of diff files
- Tile expire list for re-rendering updated areas
- Background mode for diff-import (update DB in background)
- Improve parallelization of diff import
Installation
Binary
There are no official releases, but you find development builds at http://imposm.org/static/rel/.
These builds are for x86 64bit Linux and require no further depedecies. Download, untar and start imposm3
.
(Note: These binaries require glibc >= 2.15 at the moment.
Ubuntu 12.04 is recent enough, Debian 7 not. Future binary releases will work on older versions as well.)
Source
There are some dependencies:
Compiler
You need Go >=1.1.
C/C++ libraries
Other dependencies are libleveldb, libgeos, protobuf and libsqlite3. Imposm 3 was tested with recent versions of these libraries, but you might succeed with older versions. GEOS >=3.2 is recommended, since it became much more robust when handling invalid geometries. For best performance use HyperLevelDB as an in-place replacement for libleveldb.
Go libraries
Imposm3 uses the following libraries. go get
will fetch these:
- https://github.com/jmhodges/levigo
- https://code.google.com/p/goprotobuf/proto
- https://code.google.com/p/goprotobuf/protoc-gen-go
- https://github.com/lib/pq
Other
Fetching Imposm and the Go libraries requires mercurial and git.
Compile
Create a new Go workspace:
mkdir imposm
cd imposm
export GOPATH=`pwd`
Get Imposm 3 and all dependencies:
git clone https://github.com/omniscale/imposm3 src/imposm3
go get imposm3
go install imposm3
Done. You should now have an imposm3 binary in $GOPATH/bin
.
Go compiles to static binaries and so Imposm 3 has no runtime dependencies to Go.
Just copy the imposm3
binary to your server for deployment. The C/C++ libraries listed above are still required though.
Usage
imposm3
has multiple subcommands. Use imposm3 import
for basic imports.
For a simple import:
imposm3 import -connection postgis://user:password@host/database \
-mapping mapping.json -read /path/to/osm.pbf -write
You need a JSON file with the target database mapping. See example-mapping.json
to get an idea what is possible with the mapping.
Imposm creates all new tables inside the import
table schema. So you'll have import.osm_roads
etc. You can change the tables to the public
schema:
imposm3 import -connection postgis://user:passwd@host/database \
-mapping mapping.json -deployproduction
You can write some options into a JSON configuration file:
{
"cachedir": "/var/local/imposm3",
"mapping": "mapping.json",
"connection": "postgis://user:password@localhost:port/database"
}
To use that config:
imposm3 import -config config.json [args...]
For more options see:
imposm3 import -help
Sorry, that's all documentation for the moment.
Note: TLS/SSL support is disabled by default due to the lack of renegotiation support in Go's TLS implementation. You can re-enable encryption by setting the PGSSLMODE
environment variable or the sslmode
connection option to require
or verify-full
, eg: -connect postgis://host/dbname?sslmode=require
. You will need to disable renegotiation support on your server to prevent connection errors on larger imports. You can do this by setting ssl_renegotiation_limit
to 0 in your PostgreSQL server configuration.
Support
There is a mailing list at Google Groups for all questions. You can subscribe by sending an email to: imposm+subscribe@googlegroups.com
For commercial support contact Omniscale.
Development
The source code is available at: https://github.com/omniscale/imposm3/
You can report any issues at: https://github.com/omniscale/imposm3/issues
Test
Unit tests
go test imposm3/...
System tests
There is a system test that imports and updates OSM data and verifies the database content.
This test is written in Python and requires nose
, shapely
and psycopg2
. You also need osmosis
to create test PBF files.
There is a Makefile that (re)builds imposm3
and creates all test files if necessary and then runs the test itself.
make test
WARNING: It uses your local PostgeSQL database (import
schema), if you have one. Change the database with the standard PGXXX environment variables.