Windows support

[Julian Simioni]

The pelias/docker setup does not currently officially support Windows.

No one on the Pelias core team regularly uses Windows, and there are many different varieties of ways to use Docker with Windows, so it's a large task to ensure compatibility.

If you have familiarity with using Docker on Windows platforms and would like to help us improve Windows support, let us know.

[michalstepien]

It works nice on Windows. Step by step.

1. Share drive:
   

2. Increase memory:

3. Create .env file, for example:

COMPOSE_PROJECT_NAME=pelias
DOCKER_USER=1000
DATA_DIR=D:\project\PortaAngusta.Pelias\docker\projects\poland
WOF_DIR=D:\project\PortaAngusta.Pelias\docker\projects\poland\whosonfirst\data

3. Run commands:

docker-compose run --rm whosonfirst ./bin/download
docker-compose run --rm openaddresses ./bin/download
docker-compose run --rm openstreetmap ./bin/download
docker-compose run --rm geonames ./bin/download
docker-compose run --rm interpolation ./bin/download-tiger
docker-compose run --rm transit ./bin/download
docker-compose run --rm csv-importer ./bin/download

4. If you dont want download whosonfirst all countries (6GB) , go to: https://geocode.earth/data/whosonfirst/ and choose your country. Unzip to WOF_DIR

5. Run commands:

docker-compose run --rm polylines bash ./docker_extract.sh;
docker-compose run --rm interpolation bash ./docker_build.sh;
docker-compose run --rm placeholder ./cmd/extract.sh;
docker-compose run --rm placeholder ./cmd/build.sh;

[jeffreyptruong]

I used the quickstart build script (https://github.com/pelias/docker#quickstart-build-script) as the basis to get it working on Windows. I looked into the pelias helper script and the scripts in the cmd folder for the docker functions and executed them through the Windows PowerShell. I also used @michalstepien method for whosonfirst (WOF) as the WOF download script could not extract the SQLite db @orangejulius @missinglink. This is experimental but here are the steps I used to get the Australian build up and running:

Setting up

1. Clone the pelias docker repo. In my example I cloned it to C:\Users\xxxxxxxx\DockerProjects\pelias
   
   The cloned repo creates a top level folder called “docker”. My preference was to remove this folder level resulting in:
   

2. Open the .env file located in C:\Users\xxxxxxxx\DockerProjects\pelias\projects\australia then edit the line
   DATA_DIR=/tmp/pelias/australia to DATA_DIR=C:\Users\xxxxxxxx\DockerProjects\pelias\projects\australia
   
   

Building pelias

This part took my machine several hours to complete.

1. Open PowerShell and change the working directory to the Australia folder in the pelias project folder
   cd C:\Users\xxxxxxxx\DockerProjects\pelias\projects\australia
2. Pull all pelias related images for build
   docker-compose pull
3. Start elasticsearch
   docker-compose up -d elasticsearch
4. Create the pelias index for elasticsearch. Before executing the code below make sure that the elasticsearch instance has a green status by checking the following address localhost:9200/_cat/health
   If the status is green then run the following to generate the index
   docker-compose run --rm schema ./bin/create_index;
5. The next part downloads all the sources of address and related geographic files to power pelias
   docker-compose run --rm whosonfirst ./bin/download;
docker-compose run --rm openaddresses ./bin/download;
docker-compose run --rm openstreetmap ./bin/download;
docker-compose run --rm interpolation ./bin/download-tiger;
   
   As mentioned before WOF will fail. I went to the WOF folder where the whosonfirst-data-latest.db.bz2 was downloaded to and extracted it to the same directory of the downloaded file.
   
   
   I also downloaded the WOF bundles from https://geocode.earth/data/whosonfirst for Australia then extracted the data and meta folder into the WOF folder.
   
   
   At this stage, this is the only method I can find to get WOF working for Windows.
6. Prepare data for polylines, placeholder, and interpolation
   docker-compose run --rm polylines bash ./docker_extract.sh;
docker-compose run --rm placeholder ./cmd/extract.sh;
docker-compose run --rm placeholder ./cmd/build.sh;
docker-compose run --rm interpolation bash ./docker_build.sh;
7. Import data
   docker-compose run --rm whosonfirst ./bin/start;
docker-compose run --rm openaddresses ./bin/start;
docker-compose run --rm openstreetmap ./bin/start;
docker-compose run --rm polylines ./bin/start;
   
   I've tried this method 3 times on a different machines and the import seems to fail unless I rerun the create index function in step 4.
8. Start up pelias
   docker-compose up
   
   All of the pelias services should be available on the respective default ports. pelias seems to be functioning but have not done any tests to see if it is running correctly.

[Peter Johnson]

I'm actually really happy to see that we have developers running Pelias on Windows and that it actually works!

The pelias command is currently written in bash so I started a port to nodejs here #173 which should allow it to work cross-platform.

The pelias command itself is actually fairly simple, it just shells out most of the work to docker-compose.

I don't have a lot of time to work on it at the moment but I'd be happy if you'd like to pick up where I left off and we could work together to replace the bash version with one which works equally well on nix/mac/win?

[jeffreyptruong]

@missinglink I'm happy to help where I can. However, this sort of work is outside my main domain as I'm mostly in the data science space. I've happened to pick up on this as we are looking into alternatives for geocoding sensitive data and that we run mainly a Windows stack.

At this stage I think the main failing component on Windows during my testing is the whosonfirst downloader. I'm not sure if it is related to permissions but the downloader does download the whosonfirst-data-latest.db.bz2 file but seems to fail to decompress it:

[whosonfirst-sqlite-decompress]  /data/whosonfirst/sqlite/whosonfirst-data-latest.db.bz2
bunzip2: Can't create output file /data/whosonfirst/sqlite/whosonfirst-data-latest.db: No such file or directory.
error downloading: whosonfirst-data-latest.db

/code/pelias/whosonfirst/node_modules/better-sqlite3/lib/database.js:45
        return new CPPDatabase(filename, filenameGiven, memory || anonymous, readonly, fileMustExist, timeout, verbose || null);
               ^
SqliteError: unable to open database file
    at new Database (/code/pelias/whosonfirst/node_modules/better-sqlite3/lib/database.js:45:9)
    at findSubdivisions (/code/pelias/whosonfirst/utils/sqlite_extract_data.js:195:12)
    at /code/pelias/whosonfirst/utils/download_data.js:19:26
    at /code/pelias/whosonfirst/node_modules/async/dist/async.js:2959:19
    at wrapper (/code/pelias/whosonfirst/node_modules/async/dist/async.js:272:20)
    at iterateeCallback (/code/pelias/whosonfirst/node_modules/async/dist/async.js:425:28)
    at /code/pelias/whosonfirst/node_modules/async/dist/async.js:325:20
    at /code/pelias/whosonfirst/node_modules/async/dist/async.js:2957:17
    at ChildProcess.<anonymous> (/code/pelias/whosonfirst/utils/sqlite_download.js:45:9)
    at ChildProcess.emit (events.js:209:13)

I'll need to edit my guide above as I actually did a workaround by using a hybrid approach to get WOF working. For the placeholder component I used the WOF bundles I downloaded from https://geocode.earth/data/whosonfirst. As for the importing of WOF into elasticsearch I used the sqlite approach by setting sqlite parameter as true under the whosonfirst block in the pelias.json file.

[Peter Johnson]

One thing which stood out to me as unusual from the original comment was that the WOF_DIR= env var was set, we don't usually set this as Linux installs only require DATA_DIR=.

Does setting that fix the WOF download issue?

[jeffreyptruong]

I will need to test it again but I recall that without setting WOF_DIR I still get the same error. I'll also need to do a build on my home pc as I've been testing this on my work pc to rule out any issues with admin rights.

[Jim | OrbitalGun]

Windows 10 v2004 is scheduled for release in May. It includes a new version of Windows Subsystem for Linux(WSL2). This will enable access to Docker in WSL.

I installed the Windows Insider build, and I was able to use the Quickstart script to install the portland-metro project. The only problem was with the WOF install, but that looks like an issue with the WOF importer.

[Sebastian Cepeda]

I've tried both options, and the one with the docker running inside of the ubuntu18.04, therefore using the normal commands, was the best.

[Anderson Carlos Ferreira da Silva]

Hello,

I just want to share my solution to run on Windows (Docker/WSL2):

#!/bin/bash
set -x

# change directory to the where you would like to install Pelias
# cd /path/to/install

# clone the repository
if [ ! -d "./docker" ]; then
	git clone https://github.com/pelias/docker.git
fi

# cd into the project directory
cd docker/projects/portland-metro

# create a directory to store Pelias data files
# see: https://github.com/pelias/docker#variable-data_dir
# note: use 'gsed' instead of 'sed' on a Mac
mkdir -p ./data
sed -i "/DATA_DIR/d" .env
echo DATA_DIR=$(pwd -W)/data >> .env

# check the docker daemon
# see: https://stackoverflow.com/questions/43721513/how-to-check-if-the-docker-engine-and-a-docker-container-are-running
# see: https://stackoverflow.com/questions/3466166/how-to-check-if-running-in-cygwin-mac-or-linux
if [ ! docker info > /dev/null 2>&1 ]; then
	echo "The docker docker daemon may be not running!"
fi

# add pelias script to the path
export PATH=$PATH:$(readlink -m $(pwd)/../../)

# run build
if [ $(docker container inspect pelias_libpostal --format "{{.Config.Image}}") = "pelias/libpostal-service" ] && 
   [ $(docker container inspect pelias_api --format "{{.Config.Image}}") = "pelias/api:master" ] && 
   [ $(docker container inspect pelias_placeholder --format "{{.Config.Image}}") = "pelias/placeholder:master" ] && 
   [ $(docker container inspect pelias_interpolation --format "{{.Config.Image}}") = "pelias/interpolation:master" ] && 
   [ $(docker container inspect pelias_pip-service --format "{{.Config.Image}}") = "pelias/pip-service:master" ] && 
   [ $(docker container inspect pelias_elasticsearch --format "{{.Config.Image}}") = "pelias/elasticsearch:7.16.1" ]; then
	pelias elastic start
	pelias elastic wait	
	pelias compose up
else
	pelias compose pull
	pelias elastic start
	pelias elastic wait
	pelias elastic create
	pelias download all
	pelias prepare all
	pelias import all
	pelias compose up

	# optionally run tests
	pelias test run
fi

Basically, it is the same Quickstart build script. Besides, it is possible to change the path permanently...

Have a good time,

PS : it was tested on PowerShell and Mingw-w64 with Docker Desktop/WSL2 (Windows 10)