Giters

schlomm / koop

General repo for koop

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

#Koop - Custom Setup

Major parts of this documentation might be outdated as koop has experienced many changes!!!

Information about koop can be found here. In short: koop is something like an adapter between several (mapping-)APIs to serve those as ESRI Feature Service or as a geojson output. You can also describe it as a small "Feature Manipulation Service". For more information check the above link.

Preparing and installing the system

Add repositories and sources and update
  1. sudo add-apt-repository ppa:ubuntu-toolchain-r/test
  2. sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt trusty-pgdg main" >> /etc/apt/sources.list
  3. wget --quiet -O - http://apt.postgresql.org/pub/repos/apt/ACCC4CF8.asc | sudo apt-key add -
  4. sudo apt-get update
  5. sudo apt-get install
Install needed software (g++ compiler, nodejs, git, postgresql, postgis)
  1. sudo apt-get install software-properties-common
  2. sudo apt-get install g++
  3. sudo apt-get install libpq-dev
  4. sudo apt-get install nodejs
  5. sudo apt-get install nodejs build-essential
  6. sudo apt-get install git
  7. sudo apt-get install postgresql-9.3-postgis-2.1 postgresql-contrib
Configure postgresql
  1. sudo su -l postgres
  2. postgres@ubuntu-14:~$ pg_createcluster 9.3 main --start
  3. postgres@ubuntu-14:~$ exit
  4. sudo service postgresql start
  5. sudo -u postgres psql postgres

Change password for 'postgres'-user and enable PostGIS Extension (can also be done via pgAdmin)

  1. \password postgres
  2. CREATE EXTENSION POSTGIS
Database configuration

Database connection to the world (of course only if you want to have this publicly available database)

  1. nano /etc/postgresql/9.3/main/pg_hba.conf
  2. Place the following line: "host all all 0.0.0.0/0 md5" in the corresponding code-block (somewhere at the end of the file) and save the changes.
  3. sudo nano postgresql.conf
  4. Change the "CONNECTIONS AND AUTHENTICATION" part to : listen_addresses = '*' (remove # and paste *)
  5. Restart postgresSQL database: sudo /etc/init.d/postgresql restart

koop preparations & setup

Koop needs some individual modifications and edits, to set it up for your local and personal environment.

Download

Clone the koop-repo to your server:

  1. git clone https://github.com/Esri/koop.git
  2. cd koop/
koop-providers

Edit the providers.json, so custom providers also be installed by executing setup.js. A custom providers.json can look like the following

--------
{
  "koop-gist": "https://github.com/chelm/koop-gist/tarball/master",
  "koop-github": "https://github.com/chelm/koop-github/tarball/master",
  "koop-agol": "https://github.com/Esri/koop-agol/tarball/master",
  "koop-socrata": "https://github.com/chelm/koop-socrata/tarball/master",
  "koop-ckan": "https://github.com/chelm/koop-ckan/tarball/master",
  "koop-acs": "https://github.com/chelm/koop-acs/tarball/master",
  "koop-dwd": "https://github.com/schlomm/koop-dwd/tarball/master",
  "koop-ckan_govdata": "https://github.com/schlomm/koop-ckan_govdata/tarball/master",
}
--------
  1. Execute the setup.js via node, so the providers will be added to the package.json: node setup.js
Database support

Enable Database Support for koop to cache data in PostGIS. Fot his: Set the right runtime.json in /koop/config/runtime.json

{
  "db":{
    "postgis": {
  "conn": "postgres://postgres:postgrespassword@localhost:5432/postgres"
}
  },
  "server": {
  "port": 1337
  }
}

### where "postgis" is the database type (so the extension has to be enabled first)
### where "postgres://postgres:postgrespassword@localhost:5432/postgres"  is -> 'databasetype'://user:databasepassword@host:port/databasename
Install koop-application

Within the koop dir, install koop

  1. cd /home/koop
  2. npm install

The installation can take several minutes.

Koop-Github Gist Module configuration:

  1. For using koop-gist you need to have a Github Token, which has to be generated in Github under you Account preferences. This has to be stored in the models-dir of the koop-gist-module.
  2. Open and Edit the config-file: nano koop/node_modules/koop-gist/models/config.js.example
  3. Add your API Key: exports.token = "token here";
  4. Save file under config.js or simply copy it via cp config.js.example config.js
Examples:

Koop-Github Module configuration:

  1. For using koop-github you need to have a Github Token, which has to be generated in Github under you Account preferences. This has to be stored in the models-dir of the koop-gist-module.
  2. Open and Edit the config-file: nano koop/node_modules/koop-github/models/config.js.example
  3. Add your API Key: exports.token = "token here";
  4. Save file under config.js or simply copy it via cp config.js.example config.js
Examples:

Koop-Socrata setup and configuration:

Install koop-socrata Extension only if it is not already installed (check koop/node_modules/ beforehand) or if it is not installed by default using a customized providers.json (check koop-setup preparations section

  • npm install https://github.com/chelm/koop-socrata/tarball/master
Register a socrata-Provider

Please note that the koop-server has to be up and running). Some example socrata portals.

  • curl --data "host=https://data.nola.gov&id=nola" localhost:1337/socrata
  • curl --data "host=https://data.seattle.gov&id=seattle" localhost:1337/socrata
  • curl --data "host=https://data.colorado.gov&id=colorado" localhost:1337/socrata
Use koop-socrata
  • Get a list of all registered socrata providers: localhost:1337/socrata/
  • Get more information about a specific provider by specifying the provider-id: localhost:1337/socrata/<id>
  • Access socrata dataset as raw geojson: localhost:1337/socrata/<socrataportal>/<datasetid>
  • Access socrata dataset as ESRI FeatureService: localhost:1337/socrata/<socrataportal>/<datasetid>/FeatureServer
  • Preview socrata dataset on a map: localhost:1337/socrata/<socrataportal>/<datasetid>/Preview
Examples:

Koop-CKAN setup and configuration:

Install koop-ckan Extension only if it is not already installed (check koop/node_modules/ beforehand) or if it is not installed by default using a customized providers.json (check koop-setup preparations section

  • npm install https://github.com/chelm/koop-ckan/tarball/master
Register a ckan-Provider

Please note that the koop-server has to be up and running). Some example socrata portals.

  • ``curl --data "host=https://catalog.data.gov&id=datagov" localhost:1337/ckan`
  • curl --data "host=https://data.hdx.rwlabs.org&id=rwlabs" localhost:1337/ckan
  • curl --data "host=https://offenedaten.de&id=offenedaten" localhost:1337/ckan
  • curl --data "host=http://suche.transparenz.hamburg.de&id=hamburg" localhost:1337/ckan
  • curl --data "host=http://demo.ckan.org&id=demockan" localhost:1337/ckan
  • curl --data "host=http://data.kk.dk/&id=datakk" localhost:1337/ckan

Please note that ckan-portals and their datasets have to follow the following structure:

  • portalURL/api/3/action/package_list
  • portalURL/api/3/action/package_show?id=datasetid
  • portalURL/api/3/action/package_search or /api/3/action/package_search?q=datasetid
Use koop-ckan
  • Get a list of all registered ckan providers: localhost:1337/ckan/
  • Get more information about a specific provider by specifying the provider-id: localhost:1337/ckan/<id>
  • Access ckan dataset as raw geojson: localhost:1337/ckan/<ckanportal>/<datasetid>
  • Access ckan dataset as ESRI FeatureService: localhost:1337/ckan/<ckanportal>/<datasetid>/FeatureServer
  • Preview ckan dataset on a map: localhost:1337/ckan/<ckanportal>/<datasetid>/Preview

Koop-AGOL (ArcGIS Online) setup and configuration:

Install koop-agol extension only if it is not already installed (check koop/node_modules/ beforehand) or if it is not installed by default using a customized providers.json (check koop-setup preparations section

  • npm install https://github.com/chelm/koop-agol/tarball/master
Register a agol-Provider

Please note that the koop-server has to be up and running). Some example socrata portals.

  • curl -i -X POST -d "id=arcgis" -d "host=https://www.arcgis.com" localhost:1337/agol
Use koop-agol
  • Get a list of all registered agolproviders: localhost:1337/agol/
  • Get more information about a specific provider by specifying the provider-id: localhost:1337/agol/<id>
  • Access agol dataset as raw geojson: localhost:1337/agol/<agolportal>/<datasetid>
  • Access socrata dataset as ESRI FeatureService: localhost:1337/agol/<agolportal>/<datasetid>/FeatureServer
  • Preview socrata dataset on a map: localhost:1337/agol/<agolportal>/<datasetid>/Preview
Examples:

Other koop-proviers setup and configuration

If you want to use other available providers, it's mainly the same as the above mentioned. Some do not need to have the registering-procedure, because the are portal specific.

Using koop

  • Start node server: node server.js , where output is loged in current terminal session
  • Start node server and keep it up and running: nohup node server.js > output.log &', where output is logged to output.log and where the server is not terminated when terminal is closed or looses connection. You can also press CTRL+C
  • Kill Server:
    • If node server.js was used: CTRL+C
    • If nohup node server.js > output.log &, you cann kill the node.js server by knowing the running port:
      1. sudo fuser -v 1337/tcp - gives you the process running on port 1337
      2. sudo fuser -vk 1337/tcp - kills all running processes on port 1337
  • List all installed providers: localhost:1337/providers
  • Delete ckan:services via SQL: DELETE FROM "ckan:services" WHERE id = 'govdatadaten' OR id = 'govdatanew';

Error & Fixes

If there is something to to with the default.yml, you need to install the js-yaml module for nodejs in the "koop/node_modules" folder via npm install js-yaml. More infos here: https://www.npmjs.org/package/js-yaml

About

General repo for koop