NISMOD v2
NISMOD (National Infrastructure Systems Model) is an integrated model of infrastructure systems, developed as part of the ITRC/MISTRAL project.
NISMOD v2 has several components:
- released versions of infrastructure sector models
- an integration framework, smif
- smif configuration to link the models into a system-of-systems model
- smif-compatible scenario, narrative and planning data to run linked model runs
- (for some models) internal model data, used by the model but not exposed to smif as part of the system-of-systems configuration
This repository contains only:
- smif configuration needed to link the models into a system-of-systems and set up model runs
- model wrappers and various helpers to set up runs and explore results.
In particular:
- prepared model data is not yet provided (references to Amazon S3 buckets below are not public)
- models and dependencies must be installed separately, following instructions below (water and energy supply models are not open; transport and digital models are open)
Setup
To set up NISMOD, first clone this repository, or download a release. The directories in this folder contain or will contain everything required for a model run:
config
: model configurationdata
: (initially incomplete) model datamodels
: model wrappersnotebooks
: jupyter notebooks for exploring model resultsplanning
: decision modelsprovision
: scripts to install models and get dataresults
: (initially empty) for model run resultstest
: integration testsutilities
: helpful scripts to create or migrate configuration
Then in outline (more details further below) either install everything directly to run nismod on a desktop, server or cluster:
- Install smif (recommend using
conda
) - Install model dependencies:
- FICO XPRESS (for energy supply)
- Java (for transport)
- PostgreSQL and ODBC (for energy supply)
- Configure:
- connection details for the NISMOD AWS S3 bucket
s3://nismod2-data
(copyprovision/template.s3.ini
toprovision/s3.ini
and edit) - connection details for a Postgres database (copy
provision/template.dbconfig.ini
toprovision/dbconfig.ini
and edit for your database)
- connection details for the NISMOD AWS S3 bucket
- Install models (run
provision/install_*
scripts) - Get data (run
provision/get_data_*
scripts)
Or set up a virtual machine for testing:
- Check that you have VirtualBox and Vagrant installed
- Configure:
- connection details for the NISMOD AWS S3 bucket (copy
template.s3.ini
tos3.ini
and edit) - connection details for a Postgres database (copy
template.dbconfig.ini
todbconfig.ini
) - FICO XPRESS license (copy
template.xpauth.xpr
or your own license toxpauth.xpr
)
- connection details for the NISMOD AWS S3 bucket (copy
- Setup a virtual machine (run
vagrant up
then connect withvagrant ssh
)
Download NISMOD
Download the latest release of NISMOD v2.0 from GitHub as a .zip archive and unzip it into a directory.
Alternatively, clone the NISMOD v2.0 repository from https://github.com/nismod/nismod2.
cd path/to/git/projects # somewhere to keep the nismod project
git clone https://github.com/nismod/nismod2.git
cd nismod2
To update NISMOD to the latest development version:
cd path/to/nismod2 # wherever you cloned the nismod folder
git checkout master # to make sure you’re on the master branch
git pull # pull changes from Github to your local machine
Configure connection details for the NISMOD AWS S3 bucket
Within the NISMOD folder, copy provision/template.s3.ini
to provision/s3.ini
and add your
credentials to access the NISMOD AWS S3 bucket:
[profile nismod2-s3]
aws_access_key_id = <your-access-key-id>
aws_secret_access_key = <your-secret-access-key>
region = eu-west-2
output = json
Contact Tom Russell or Craig Robson to request credentials.
Upload data to the NISMOD AWS S3 bucket
This assumes the awscli
command line client is installed and
you have credentials.
pip install awscli # install
aws configure # configure (this will ask for access key and secret key)
To check contents of the bucket:
aws s3 ls s3://nismod2-data
To upload a zip:
aws s3 cp mydata_v1.1.zip s3://nismod2-data/model_name/mydata_v1.1.zip
Further usage and documentation: AWS Command Line Interface docs
Configure connection details for a Postgres database
Within the NISMOD folder, copy provision/template.dbconfig.ini
to provision/dbconfig.ini
and edit to connect to your database.
If you intend to use the vagrant virtual machine with the database as provisioned by default, leave the details unchanged.
[energy-supply]
dbname=vagrant
host=localhost
user=vagrant
password=vagrant
port=5432
Configure FICO XPRESS license
If you will install FICO XPRESS manually, skip this step.
Within the NISMOD folder, copy provision/template.xpauth.xpr
to provision/xpauth.xpr
.
If you are running within the University of Oxford OUCE network, the license server should be available and no changes are necessary. Otherwise, set the value to connect to your local license server or replace this file with a license file provided by FICO support (contact via https://www.fico.com)
use_server server="ouce-license.ouce.ox.ac.uk"
Running NISMOD on desktop/server/cluster
To run NISMOD directly, you will need to install smif, model dependencies and models.
The provision/vm_provision.sh
script may be a useful point of reference, as it is run by the
root user within the virtual machine (on Ubuntu 16.04) to automate the same process.
Install smif
See installation instructions for full details.
The recommended method uses conda
):
conda config --add channels conda-forge # enable the conda-forge channel, required for smif
conda create --name nismod python=3.6 smif # create a conda environment for nismod
conda activate nismod # activate the nismod conda environment
Install model dependencies
FICO XPRESS (for energy supply)
See energy supply documentation for full details.
On Linux (x64) it should be possible download and install FICO by running:
mkdir xpress_setup
wget -nc -qO- \
"https://clientarea.xpress.fico.com/downloads/8.5.10/xp8.5.10_linux_x86_64_setup.tar" \
--no-check-certificate | tar -C xpress_setup -xv
./xpress_setup/install.sh
Java (for transport)
See transport model documentation for full details.
To run the transport model, it should be sufficient to have the Java Runtime Environment (JRE) version 8 installed.
On Ubuntu, run:
sudo apt update && sudo apt install default-jre
PostgreSQL and ODBC (for energy supply)
See energy supply documentation for full details.
The energy supply model connects to a Postgres database via ODBC, so requires access to a Postgres server, an ODBC installation and the Postgres ODBC drivers.
On Ubuntu, install Postgres, libpq (shared library for postgres clients), ODBC and postgres connector:
sudo apt update && sudo apt install \
postgresql postgresql-contrib \
libpq-dev \
unixodbc-dev odbc-postgresql
Get data
Install boto3
: pip install boto3
.
Run each of the provision/get_data_*
scripts:
bash ./provision/get_data_digital_comms.sh .
bash ./provision/get_data_energy_demand.sh .
bash ./provision/get_data_energy_supply.sh .
bash ./provision/get_data_transport.sh .
bash ./provision/get_data_water_supply.sh .
bash ./provision/get_data_water_demand.sh .
Install models
Run each of the provision/install_*
scripts:
bash ./provision/install_digital_comms.sh .
bash ./provision/install_energy_demand.sh .
bash ./provision/install_energy_supply.sh . ./path/to/xpress_install_dir
bash ./provision/install_transport.sh .
bash ./provision/install_water_supply.sh .
bash ./provision/install_water_demand.sh .
Run post-install scripts:
energy_demand setup -f ./models/energy_demand/wrapperconfig.ini
Note that install_water_supply.sh
clones a repository that is currently private: setting up the water supply model may require requesting access to the repository.
Running NISMOD on a virtual machine
The Vagrantfile defines the automated setup for a virtual machine, which will provide a reproducible development environment.
In virtual machine terminology, the virtual machine (vm for short) is sometimes called the ‘guest’ machine and the physical computer is called the ‘host’ machine.
To use the NISMOD virtual machine, first install:
If you see only 32-bit options in Virtualbox, please ensure that:
- Hardware virtualization is enabled in the BIOS
- For Intel x64: VT-x (Intel Virtualization Technology) and VT-d are both enabled
- For AMD x64: AMD SVM (Secure Virtual Machine) is enabled
- For Windows: in Windows Features, "Hyper-V platform" is disabled.
Note for Ubuntu 17.10 users: If you are experiencing the issue The box ‘bento/ubuntu-16.04’ could not be found or could not be accessed in the remote catalog. Make sure that you have the latest version of Vagrant (>v2) installed. This version is currently not in the standard package archive (PPA) but can be downloaded from the vagrant website.
Create and configure the guest machine, running this command from the NISMOD directory:
vagrant up
This will download a virtual machine image, install all the packages and software which are required to test and run NISMOD onto that virtual machine and download the data and model releases from the AWS S3 bucket.
Once it has run, you should be able to:
vagrant ssh # log in to the virtual machine on the command line
cd /vagrant # move to the folder that’s shared between the host and guest machines
ls # list files and folders
smif list # list available model runs
smif run energy_supply_toy # to run the energy_supply_toy model run
logout # log out of the virtual machine (or shortcut: CTRL+D)
Then results are written to subfolders within the results
directory.
Running NISMOD with smif binary (parquet) store
Parquet files are smaller, typed and faster to read and write than CSV.
Run smif csv2parquet
converter after getting data and installing models, or after updates.
smif csv2parquet --noclobber data
Then run model-runs using the local_binary
interface. smif will read and write parquet
files.
smif run et_module_test --interface local_binary
Updating
To reload and re-provision the virtual machine after updating NISMOD:
vagrant reload --provision
Accessing services from outside the virtual machine
It is possible to connect to some services running in the guest virtual machine directly from the host. Vagrant is set to automatically forward some ports.
Viewing Results
NISMOD includes some experimental jupyter notebooks to help access and view results.
First, start the notebook server in the background with the command:
vagrant ssh
cd /vagrant
jupyter notebook &> /dev/null &
Then navigate in a browser to
localhost:8910
Database Connection
Some sector models use a database which runs within the virtual machine and is set up when the virtual machine is provisioned.
The virtual machine exposes the database on port 6543 on the host machine, so you should be able to connect with the following details:
key | value |
---|---|
Host | localhost |
Database | vagrant |
User | vagrant |
Password | vagrant |
Port | 6543 |
For example, on the command line, assuming the psql
postgres command line client is installed
on the host machine, you should be able to connect with:
psql -d vagrant -h localhost -U vagrant -p 6543
Security note
This database setup is intended for testing and development only and assumes that high-numbered ports are only accessible to the local user. Strong passwords and secure login methods should be used if the host machine is expected to be accessed by multiple users.
Development
General Guidelines
master
branch is protected against push and force-push.- The
master
branch should be treated as live and is used for major releases only - All development occurs on the
develop
branch - New features are never added to an existing release branch
For further detail, see this article
Developing a new feature
- Create a new feature branch from the develop branch
git checkout -b feature/<feature_name> develop
- Develop your feature
- Submit a pull request against the develop branch
- Merge the changes into the develop branch
Release Process
-
Update the changelog under the heading Version X.Y.Z
-
Check versions of models and data specified in
provision/config.ini
-
Now, create a branch from develop e.g.
checkout -b release-x.y develop
-
Tag this as a release candidate
git tag -a vx.y.z-rc1
-
Any bugfixes are committed to this branch and should be tagged with incremented release candidate tags e.g.
git tag -a vx.y.z-rc2
-
Once the release candidate is stable, submit a pull request to the
master
branch -
Merge the pull request and tag
git tag -a vx.y.z
Fixing a Bug in a release
- Create a branch from master
git checkout -b hotfix/<bug_name> master
- Fix the bug by committing to the branch
- Submit a pull request to master and tag the fix
git tag -a vx.y.z
incrementing the minorz
digit
Testing
The tests in this repository are intended to test the integration of the various
NISMOD sector models with smif
, the simulation modelling integration
framework. In outline, tests here should:
- validate possible system-of-systems model configurations
- run each sector model through
smif
with only scenario dependencies (not coupling with any other models)- without unexpected error
- with the expected type contract of results
- run combinations of sector models
- without unexpected error
- with the expected type contract of results
- (optionally) with regression tests against known-good modelled outputs
Continuous integration testing is currently available through a GitLab server based in the School of Geography in Oxford. You can access the project on OUCE GitLab and monitor jobs through the web interface.
To add the remote and push a branch for automated testing, run:
git remote add ouce https://gitlab.ouce.ox.ac.uk/NISMOD/nismod2.git
git checkout feature/branch
git push -u ouce
Gitlab-runner
There is a pool of gitlab-runner workers active that pick up and run jobs on each commit/push. A local gitlab-runner can be setup using the following configuration.
concurrent = 1
check_interval = 0
[[runners]]
name = "ubuntu-nismod2"
url = "https://gitlab.ouce.ox.ac.uk/"
token = "12964bfcd398c966ebf128f1b5b34f"
executor = "docker"
[runners.docker]
tls_verify = false
image = "ubuntu:xenial"
privileged = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]
Acknowledgements
NISMOD was written and developed by researchers in the Infrastructure Transition Research Consortium as part of the EPSRC sponsored MISTRAL programme.