A pilot to display how cool geographic time-lapse data can be, when served with vector tiles and dynamic client-side filtering is used.
https://github.com/OpenHistoricalMap/VectorTilesDemo
https://openhistoricalmap.github.io/VectorTilesDemo/
- Tile server: Amazon S3, static PBF files generated by Tippecanoe
- Map client: Mapbox GL JS API
- Client-side programming language: jQuery
- Server-side programming language: none, static HTML document and S3 tiles
The generated vector tiles are served from Amazon S3; see the VECTILES_BASE_URL constant for the URL. This bucket is owned by the openhistoricalmap Organizational Unit, and relevant credentials will be in 1Password, or available from the client.
The website is served from Github Pages (URL above). A custom domain name is still pending.
Historical data:
- Downloaded US States/Territories and Counties datasets from Newberry Library. http://publications.newberry.org/ahcbp/downloads/united_states.html
- In both shapefiles, updated the
ENDfield (date this area stopped existing, usually in favor of a new area) in regards to a region being still active/valid in the present day. The value "2000/12/31" was being used to indicate that a boundary is still in effect in the present day, which runs afoul of common sense and date filters. This was replaced with "9999/12/31" to indicate still-present. - Opened both shapefiles and massaged attributes to have friendlier names (NAME instead of NAME_FULL) and to omit some fields we wouldn't use (start/end times as Unix timestamp, abbreviated names).
Modern boundaries:
- Downloaded County and State boundaries from ESRI.
- Removed attributes to have only NAME and STATE (for counties) or just NAME (for states).
Converted shapefiles to GeoJSON.
rm -f states-historical.geojson
ogr2ogr -t_srs epsg:4326 -f GeoJSON states-historical.geojson US_HistStateTerr_Gen0001.shp
rm -f counties-historical.geojson
ogr2ogr -t_srs epsg:4326 -f GeoJSON counties-historical.geojson US_HistCounties_Gen0001.shp
rm -f states-modern.geojson
ogr2ogr -t_srs epsg:4326 -f GeoJSON states-modern.geojson states-modern.shp
rm -f counties-modern.geojson
ogr2ogr -t_srs epsg:4326 -f GeoJSON counties-modern.geojson counties-modern.shp
Converted to on-disk vector tiles using Tippecanoe.
rm -rf states-historical
tippecanoe --output-to-directory=./states-historical -l states --minimum-zoom=3 --maximum-zoom=10 --no-tile-compression --simplification=5 --detect-shared-borders states-historical.geojson
rm -rf counties-historical
tippecanoe --output-to-directory=./counties-historical -l counties --minimum-zoom=6 --maximum-zoom=10 --no-tile-compression --simplification=5 --detect-shared-borders counties-historical.geojson
rm -rf states-modern
tippecanoe --output-to-directory=./states-modern -l states --minimum-zoom=3 --maximum-zoom=10 --no-tile-compression --simplification=5 --detect-shared-borders states-modern.geojson
rm -rf counties-modern
tippecanoe --output-to-directory=./counties-modern -l counties --minimum-zoom=6 --maximum-zoom=10 --no-tile-compression --simplification=5 --detect-shared-borders counties-modern.geojson
Sizes and metrics (z6+):
- Shapefiles after attribute pruning: State = 68 MB, County = 384 MB
- GeoJSON files: State = 190 MB, County = 1.1 GB
- Tile counts: State = 22361 tiles, County = 22317 tiles
- Tile data volume: State = 128 MB, County = 301 MB
Notes and warnings:
- Data volume and further separation.
- Tippecanoe refuses to create a vector tile larger than 500 KB in size. Even if this could be overridden, it would be infeasible to have a map client download many simultaneous tiles of this size or larger.
- To reduce the data volume, simplification is performed on the shapes. However, attribute data for thousands of historical counties and districts is still voluminous.
- In the case of county data, at zoom level 5 and further out, vector tiles cannot be created with the data as-given without deleting some features (typically, the smallest features are chosen first). However, the visual effect of this is unpleasant: at zoom 5 on a large display, a significant portion of the eastern seaboard is displayed and exhibits a large gap where many counties are omitted. These may be the smallest counties (as is typical for the east coast of the USA) but they form visible gaps and in this case some of the visible gaps are very large. These are not an artifact of client-side date filtering, but of the data being omitted from the vector tileset entirely in order to bring it into size compliance.
- The most expedient workaround was to start at z6 and to bring in some simplification, but at this scale some tiles barely slip under the 500 KB limit. For a larger situation in which one expects to be adding new data, new strategies would need to be created to "partition" the data into multiple sources, then display these multiple sources together as if they were one. (Example, segregate historical counties into 20 separate sources, each one representing a 25-year slice of time and containing only attributes active during that time slice.)
Babel, SASS/SCSS, Webpack.
Upon initial setup on your system, run nvm use and yarn install to set up build tools.
Then, your edits would be made to index.src.html index.src.scss index.src.js as the main files, and to webpack.entrypoint.js as the entry point which lists assets to include.
Running npm run build will compile the browser-side "deliverables" index.html index.css index.js via webpack. Note that these outputs are included in version control, so they may be hosted via Github Pages without us needing to work in additional tooling.
npm run serve will run a HTTP server, as well as watching and rebuilding (below). The vector tiles are considered data from a cross-origin perspective, and XHR won't work over file:// URLs, so this is important.
npm run watch will watch for changes and recompile and reload when changes are made. This is not a replacement for a final npm run build for deployment.