| Branch | Status |
|---|---|
| develop |  |
| master |  |
- Table of content
- Summary of the project
- How to run locally
- Service configuration
- GetTile
- GetCapabilities
- OpenAPI
This application translates WMTS requests into WMS requests and optionally caches the responses in S3 (by default this second level caching is disable). It uses gatilegrid module to perform this translation. It also serve the WMTS GetCapabilities as defined in the OGC Standard.
The Make targets assume you have
- bash
- python3.7
- python3.7-venv
- python3.7-dev
- pipenv
- gcc
- libpq-dev
- curl
- docker
- docker-compose
installed.
First, you'll need to clone the repo
git clone git@github.com:geoadmin/service-wmts
Then, you can run the setup target to ensure you have everything needed to develop, test and serve locally
make setup
Then you need to run some local containers (DB, WMS-BOD)
docker-compose up
That's it, you're ready to work.
To run locally enter
make serve
or the following to use gunicorn as web server:
make gunicornserve
or the following to run the service in a docker image locally:
make dockerrun
To run the unittest enter:
make test
To run only a single test module/class/method enter:
pipenv shell # activate the virtual environment
nose2 -c tests/unittest.cfg -t ${PWD} tests.unit_tests.test_functional
To test the localhost server you can use
curl http://localhost:5000/1.0.0/inline_points/default/current/4326/15/34136/7882.png
You can then check with the minio browser that a file has been saved on the S3 cache; http://localhost:9001 (use the credentials from minio.env)
To build a local docker image tagged as service-wmts:local-${USER}-${GIT_HASH_SHORT} you can
use
make dockerbuild
To push the image on the ECR repository use the following two commands
make dockerlogin
make dockerpush
In order to have a consistent code style the code should be formatted using yapf. Also to avoid syntax errors and non
pythonic idioms code, the project uses the pylint linter. Both formatting and linter can be manually run using the
following command:
make format-lint
The openapi spec has also a linter, and the CI will check for spec errors.
make lint-spec
Formatting and linting should be at best integrated inside the IDE, for this look at Integrate yapf and pylint into IDE
NOTE: CI will failed if the code is not properly formatted
All settings can be found in app/settings.py but here below you have the most important one described.
| Variable | Default | Description |
|---|---|---|
| ENV_FILE | Configuration environment file. Note that environment variable will overwrite values from environment file. This is especially used for local development | |
| WMTS_PORT | 9000 |
Port of the service |
| LOGGING_CFG | ./config/logging-cfg-local.yml |
Logging configuration file |
| LOGS_DIR | ./logs |
Logging output directory. Only used by local logging configuration file. |
| DEFAULT_MODE | default |
Default operation mode see Operation Mode |
| UNITTEST_SKIP_XML_VALIDATION | False |
Validating Get Capabilities XML output in Unittest takes time (~32s), therefore with this variable you can skip this test. |
| FORWARED_ALLOW_IPS | * |
Sets the gunicorn forwarded_allow_ips. See Gunicorn Doc. This setting is required in order to secure_scheme_headers to work. |
| FORWARDED_PROTO_HEADER_NAME | X-Forwarded-Proto |
Sets gunicorn secure_scheme_headers parameter to {${FORWARDED_PROTO_HEADER_NAME}: 'https'}. This settings is required in order to generate correct URLs in the service responses. See Gunicorn Doc. |
| SCRIPT_NAME | '' |
If the service is behind a reverse proxy and not served at the root, the route prefix must be set in SCRIPT_NAME. |
| WMTS_WORKERS | 0 |
WMTS service number of workers. 0 or negative value means that the number of worker are computed from the number of cpu. |
| WSGI_TIMEOUT | 45 |
WSGI timeout. |
| SQLALCHEMY_POOL_PRE_PING | False | True will enable the connection pool “pre-ping” feature that tests connections for liveness upon each checkout. This will trigger a recycle of outdated, stale connections. Activating this option will help to get rid of idle connection timeout errors but has a slight influence on the performance. |
| SQLALCHEMY_ISOLTATION_LEVEL | READ COMMITTED |
affects the transaction isolation level of the database connection. |
| SQLALCHEMY_POOL_RECYCLE | 20 | this setting causes the pool to recycle connections after the given number of seconds has passed |
| SQLALCHEMY_POOL_SIZE | 20 | the number of connections to keep open inside the connection pool |
| SQLALCHEMY_MAX_OVERFLOW | -1 | the number of connections to allow in connection pool “overflow”, -1 will disable overflow. |
| GUNICORN_WORKER_TMP_DIR | None |
This should be set to an tmpfs file system for better performance. See https://docs.gunicorn.org/en/stable/settings.html#worker-tmp-dir. |
NOTE: max-age is usually used by the Browser, while s-maxage by the server cache (e.g. CloudFront cache, see CloudFront - Managing how long content stays in the cache (expiration)).
| Variable | Default | Description |
|---|---|---|
| GET_TILE_DEFAULT_CACHE | 'public, max-age={browser_cache_ttl}, s-maxage=5184000' |
Default cache settings for GetTile requests (default to 2 months). browser_cache_ttl is set to GET_TILE_BROWSER_CACHE_MAX_TTL. Note the s-maxage directive is usually overridden by the cache_ttl value from BOD. |
| GET_TILE_ERROR_DEFAULT_CACHE | 'public, max-age=3600' |
Default cache settings for GetTile error responses (default to 1 hour). |
| ERROR_5XX_DEFAULT_CACHE | public, max-age=5 |
Default cache settings for 5xx HTTP errors |
| GET_TILE_CACHE_TEMPLATE | 'public, max-age={browser_cache_ttl}, s-maxage={cf_cache_ttl}' |
GetTile cache-control header template used with the cache_ttl value if present for the layer in the BOD. The browser_cache_ttl value will be set cache_ttl or to the GET_TILE_BROWSER_CACHE_MAX_TTL value if the later is bigger. |
| GET_TILE_BROWSER_CACHE_MAX_TTL | 3600 |
Maximum value used for the GetTile Cache-Control max-age header in case of cache_ttl configured in BOD. |
| GET_CAP_DEFAULT_CACHE | 'public, max-age=3600, s-maxage=5184000' |
GetCapabilities cache-control header value (default to 2 months). |
| CHECKER_DEFAULT_CACHE | 'public, max-age=120' |
Checker cache-control header value (default to 2 minutes) |
| Variable | Default | Description |
|---|---|---|
| WMS_HOST | localhost |
Host name of the WMS service |
| WMS_PORT | 80 |
Port of the WMS service |
| BOD_DB_HOST | WMS Postgresql database hostname | |
| BOD_DB_PORT | 5432 |
WMS Postgresql database port |
| BOD_DB_NAME | WMS database name | |
| BOD_DB_USER | WMS database user name | |
| BOD_DB_PASSWD | WMS database user password |
See https://requests.readthedocs.io/en/latest/api/#requests.adapters.HTTPAdapter for description of the following variables.
| Variable | Default |
|---|---|
| WMS_BACKEND_POOL_CONNECTION | 10 |
| WMS_BACKEND_POOL_MAXSIZE | 10 |
| WMS_BACKEND_POOL_BLOCK | False |
| WMS_BACKEND_CONNECTION_MAX_RETRY | 0 |
| Variable | Default | Description |
|---|---|---|
| AWS_ACCESS_KEY_ID | AWS access key for S3 | |
| AWS_SECRET_ACCESS_KEY | AWS access secret for S3 | |
| AWS_S3_BUCKET_NAME | service-wmts-cache |
S3 bucket name used for 2nd level caching |
| AWS_S3_REGION_NAME | AWS Region | |
| AWS_S3_ENDPOINT_URL | AWS endpoint url if not standard. This allow to use a local S3 instance with minio | |
| HTTP_CLIENT_TIMEOUT | 1 |
HTTP client timeout in seconds for AWS S3 GetTile requests |
| Variable | Default | Description |
|---|---|---|
| APP_STAGING | 'prod' |
Filter the capabilities for this staging |
| LEGENDS_BASE_URL | "https://api3.geo.admin.ch/static/images/legends" |
Legend base url used in GetCapabilities |
Get Tile endpoint; /1.0.0/{layer_id}/default/{time}/{srid}/{zoom}/{col}/{row}.{extension}, returns a Web Map Tile. It uses a WMTS Configuration taken from BOD that is cached upon service starts.
For performance reason the WMTS Config needed to return a Web Map Tile, is cached once locally in Memory during the startup of the service. To update this cache, you need to restart the service.
mode=default
This mode is the default mode for the WMTS service. It is meant to be integrated with the full stack.
The steps are:
- Check if the tile is present on S3
- If yes return the S3 tile to the client
- Otherwise request the tile from the WMS server image
- If needed puts the tile in S3 after closing the TCP connection with client
- Return the WMS image to the client
mode=preview
This mode is meant to be used to test the configuration of the wms server (skipping caching in S3).
The steps are:
- Request the WMS Server image
- Return the WMS image to the client
No data parameter can be used for tile creation.
nodata=true
Returns OK if the image was successfully fetched and created. Can be used for tile generation.
Note the Tile will be put into 2nd level S3 cache if needed.
Because some tiles are very slow to generates; up to 30 seconds, those ones are also cached into a 2nd level cache on S3. Tiles are saved on S3 based on the BOD configuration; s3_resolution_max.
This cache is more deterministic as any other CDN cache (e.g. CloudFront cache).
The following endpoint alias for GetCapabilities are implemented:
/EPSG/<int:epsg>/<string:lang>/1.0.0/WMTSCapabilities.xml/EPSG/<int:epsg>/1.0.0/WMTSCapabilities.xml(lang=de)/1.0.0/WMTSCapabilities.EPSG.<int:epsg>.xml(lang=de)/1.0.0/WMTSCapabilities.xml?lang=<string:lang>&epsg=<int:epsg>(default query:lan=de&epsg=21781)
Those endpoints are using the view from the BOD service-wmts schema.
The service uses OpenAPI Specification to document its endpoints. This documentation is in the openapi.yml file. This file can be used with either Redoc or Swagger UI renderer.
To render the spec using Redoc do as follow
make serve-spec-redoc
Then open localhost:8080 in your Browser.
To render the spec using Swagger UI do as follow
make serve-spec-swagger
Then open localhost:8080/swagger in your Browser.
See also Swagger UI Installation for more info on the Swagger UI docker image.
The OpenAPI follow some rules and must be validated using make lint-spec