Tombolo Digital Connector

The Tombolo Digital Connector is an open source tool that enables users to seamlessly combine different sources of datasets in an efficient, transparent and reproducible way.

There are three particularly important parts to the Tombolo Digital Connector:

Importers
- Built-in importers harvest a range of data sources into the centralised data format. Examples include data from ONS, OpenStreetMap, NOMIS, the London Air Quality Network and the London Data Store. We welcome the creation of additional importers.
Centralised data format
- All data imported into the Tombolo Digital Connector adopts the centralised data format. This makes it easier to combine and modify data from different sources.
Recipes
- Users generate recipes with a declarative 'recipe language' to combine the data in different ways. This combination can generate new models, indexes and insights. For example, existing recipes can generate models of social isolation, calculate the proportion of an area covered by greenspace and even generate an active transport index. We welcome the creation of additional recipes.

For further information see the documentation.

Contributing

Looking to get involved? Have a look at the Open Source Community milestone where we have selected low hanging fruit for you to easily get involved and contribute.

Requirements

To get started you will need to install the requirements to run the Digital Connector.

Note: you’ll need to have administrator rights on your machine to install these - make sure that you do before you proceed.

Install the following via the link through to their installation page:

Installation Guides

Windows
macOS (Coming Soon)
Ubuntu/Debian (Coming Soon)

Quick start

This tutorial will guide you to a quick start on macOS. Installation tutorials for other operating systems will come soon.

A note about the Terminal

The Terminal application can be found in the Applications -> Utilities folder or quickly accessed through Spotlight. It is pre-installed in macOS so there is no need to install it.

You will need this application to run some of the commands of this tutorial. When you enter a command and press return/enter, the terminal will execute it and complete the task.

Make sure to press return after typing a command before you enter the next one.

Let's start

Open the Terminal. All the following steps will operate in it.

Check if you have installed the right versions for the requirements by entering each of the following commands in the Terminal.

java -version
psql --version
gradle --version
git --version

The output will look something like this:

$ java -version
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
$ psql --version
psql (PostgreSQL) 9.6.3
$ gradle --version

------------------------------------------------------------
Gradle 3.4
------------------------------------------------------------

Build time:   2017-02-20 14:49:26 UTC
Revision:     73f32d68824582945f5ac1810600e8d87794c3d4

Groovy:       2.4.7
Ant:          Apache Ant(TM) version 1.9.6 compiled on June 29 2015
JVM:          1.8.0_121 (Oracle Corporation 25.121-b13)
OS:           Mac OS X 10.11.6 x86_64

$ git --version
git version 2.10.1 (Apple Git-78)

Get the Digital Connector code to your local machine by cloning its repository.

git clone https://github.com/FutureCitiesCatapult/TomboloDigitalConnector

If successful, you will see a log similar to the below.

$git clone https://github.com/FutureCitiesCatapult/TomboloDigitalConnector  
Cloning into 'TomboloDigitalConnector'...
remote: Counting objects: 15761, done.
remote: Compressing objects: 100% (184/184), done.
remote: Total 15761 (delta 90), reused 193 (delta 49), pack-reused 15487
Receiving objects: 100% (15761/15761), 178.89 MiB | 3.04 MiB/s, done.
Resolving deltas: 100% (7647/7647), done.

Go to the Digital Connector root directory and rename the properties files. These can be done you running each of the following commands and pressing enter.
```
cd TomboloDigitalConnector
mv gradle.properties.example gradle.properties
mv apikeys.properties.example apikeys.properties
```
The previous commands will allow you to use the default project settings.

If you prefer/need you can amend the settings altering the default ones to the ones you decide.

Set up database

The following step sets up a main and a test database after starting the server. The test database is used by the tests and is cleared routinely. We use this to gain control over what is in the database when our tests are running and to avoid affecting any important data in your main database.

pg_ctl -D /usr/local/var/postgres -l /usr/local/var/postgres/server.log start
chmod +x create_db.sh
./create_db.sh

For more information or to change the default database and user settings access the file create_db.sh.

Run tests

A quick check on how everything has been set up is to run all the tests. If they are successful, it will mean that everything went fine.

Run the command in the Terminal.

gradle test

If successful the output will be as the following.

$ gradle test
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:compileTestJava UP-TO-DATE
:processTestResources UP-TO-DATE
:testClasses UP-TO-DATE
> Building 85% > :test > 50 tests completed
:test

BUILD SUCCESSFUL

Total time: 4 mins 50.919 secs

If the tests start to fail then check the PostgreSQL server is running and the requirements are properly installed by going through the previous steps.

About to be mentioned a couple of examples of what might have gone wrong in the process if the tests start failing.

uk.org.tombolo.core.AttributeTest > testUniqueLabel FAILED
    java.util.ServiceConfigurationError
        Caused by: org.hibernate.service.spi.ServiceException
            Caused by: org.hibernate.exception.JDBCConnectionException
                Caused by: org.postgresql.util.PSQLException
                    Caused by: java.net.ConnectException

uk.org.tombolo.core.AttributeTest > testWriteJSON FAILED
    java.util.ServiceConfigurationError
        Caused by: org.hibernate.service.spi.ServiceException
            Caused by: org.hibernate.exception.JDBCConnectionException
                Caused by: org.postgresql.util.PSQLException
                    Caused by: java.net.ConnectException

uk.org.tombolo.core.DatasourceTest > testWriteJSON FAILED
    java.util.ServiceConfigurationError
        Caused by: org.hibernate.service.spi.ServiceException
            Caused by: org.hibernate.exception.JDBCConnectionException
                Caused by: org.postgresql.util.PSQLException
                    Caused by: java.net.ConnectException

The former error log is launched if the server is not running and to solve it you need to run the command.

pg_ctl -D /usr/local/var/postgres -l /usr/local/var/postgres/server.log start

In case you see this other error instead, it means that you did not rename the settings files successfully.

FAILURE: Build failed with an exception.

* Where:
Build file '/TomboloDigitalConnector/build.gradle' line: 159

* What went wrong:
Execution failed for task ':test'.
> Test environment not configured. See the README.

If you see other errors, try to go back and follow the steps again.

Run the Digital Connector

Now you are all set to run a task on the Digital Connector.

The next step is to run an example to show how the digital connector combines different data sets. We’re using an example that shows the relationship between air pollution (demonstrated in this example by NO2 levels), and car and bicycle traffic in every borough in London. You can read more about this example here.

When you’ve run this example, you can expect a map that looks like this:

To get started:

Run the following command into the Terminal.

gradle runExport \
  -PdataExportSpecFile='src/main/resources/executions/examples/london-cycle-traffic-air-quality-lsoa-backoff.json' \
  -PoutputFile='~/Desktop/london-cycle-traffic-air-quality-lsoa-backoff-output.json'

You can expect it to take around 1.5 minutes to generate the output, which will be saved in the Desktop. Change the path in the command in case you want it saved elsewhere.

The output will look similar to the next content:

{
  "type":"FeatureCollection",
  "features":[
    {
      "type":"Feature",
      "geometry":{
        "type":"Polygon",
        "coordinates":[[[-0.0802,51.5069],[-0.1092,51.5099],[-0.1114,51.5098],
                        [-0.1116,51.5153],[-0.1053,51.5185],[-0.0852,51.5203],
                        [-0.0784,51.5215],[-0.0802,51.5069]]]
      },
      "properties":{
        "label":"E09000001",
        "name":"City of London",
        "Nitrogen Dioxide":81.3333333333333,
        "Bicycle Fraction":0.25473695591455 
      }
    }, ...
    ...
  ]
}

Once you have your output, you can open with a geospatial visualisation tool. For this example, we recommend QGIS, and here you can find a guide on how to use it.

We need your feedback!
If you have any issues with setting up the tool, or running the tutorial, or if you have some advice about how we can do this better, please contact us by creating an issue. Our goal is for someone to get back to you within 24 hours.

Run tests

gradle test

If you use the IntelliJ JUnit test runner, you will need to add the following to your VM Options in your JUnit configuration (Run -> Edit Configurations -> All under JUnit, and Defaults -> JUnit):

-enableassertions
-disableassertions:org.geotools...
-Denvironment=test
-DdatabaseURI=jdbc:postgresql://localhost:5432/tombolo_test
-DdatabaseUsername=tombolo_test
-DdatabasePassword=tombolo_test

Local deploy

To deploy to your local Maven installation (~/.m2 by default):

gradle install

Run Tasks

Run export

We use the Gradle task runExport to run exports. The parameters are as follows:

gradle runExport \
    -PdataExportSpecFile='path/to/spec/file.json' \
    -PoutputFile='output_file.json' \
    -PforceImports='com.className'
    -PclearDatabaseCache=true

For example, this calculates the proportion of cycle traffic received at a traffic counter relative to the total traffic in a given borough and outputs the results to the file reaggregate-traffic-count-to-la.json:

gradle runExport \
    -PdataExportSpecFile='src/main/resources/executions/examples/reaggregate-traffic-count-to-la.json' \
    -PoutputFile='reaggregate-traffic-count-to-la_output.json'

Export data catalogue

We use the Gradle task exportCatalogue to export a JSON file detailing the capabilities of the connector and explore the data catalogue.

gradle exportCatalogue -PoutputFile=catalogue.json

Start/Stop server

If you need to start or stop the server (on MacOS X), use the following commands.

# to start
pg_ctl -D /usr/local/var/postgres -l /usr/local/var/postgres/server.log start

# to stop
pg_ctl -D /usr/local/var/postgres -l /usr/local/var/postgres/server.log stop

License

MIT

When using the Tombolo or other GitHub logos and artwork, be sure to follow the GitHub logo guidelines.

gregmaya / TomboloDigitalConnector

Tombolo Digital Connector

Table of Contents:

Contributing

Requirements

Installation Guides

Quick start

A note about the Terminal

Let's start

Set up database

Run tests

Run the Digital Connector

To get started:

See also:

Run tests

Local deploy

Run Tasks

Run export

Export data catalogue

Start/Stop server

License

About

Languages