This is an extension to CKAN that provides customisations specifically for the data.gov.uk project.
The official version is available at: https://github.com/datagovuk/ckanext-dgu
Contributions from 1st March 2012 are by the Cabinet Office. It is Crown Copyright and opened up under both the Open Government Licence (OGL) (which is compatible with Creative Commons Attibution License) and the GNU Affero General Public License (AGPL) v3.0. Before 1st March 2012, contributions are Copyright (c) 2010-2012 Open Knowledge Foundation. This material is open and licensed under the GNU Affero General Public License (AGPL) v3.0.
OGL terms: http://www.nationalarchives.gov.uk/doc/open-government-licence/ AGPL terms: http://www.fsf.org/licensing/licenses/agpl-3.0.html
Related extensions that data.gov.uk use in addition are: ckanext-archiver, ckanext-harvest, ckanext-spatial, ckanext-os, ckanext-qa, ckanext-social
This extension contains a number of elements, principally:
- DGU-specific templates
- Custom dataset edit form - includes a number of custom fields such as temporal_coverage and geographic_coverage.
- Harvest Object inserted into the CKAN package view page.
- DGU-customized search - different facets etc
- Reports (integrated from ckanext-report)
- Drupal integration - Drupal login, user details copied into CKAN, comments below datasets, provide APIs to Drupal for displaying recent datasets
- Authorization customizations - display of users is tightened up
- Feedback form (below datasets)
- gov_daily - a script (for running daily) that save the database dumps for end-users (JSON/CSV) and backups (SQL).
- various other command-line utilities
ckanext.dgu.plugin:DguForm
ckanext.dgu.controllers.package:PackageController
Based on PackageController but at /dataset/* rather than /package/* and adds the delete function. Proxy for getting Drupal comments (only for running in paster). NO FORM HERE
ckanext.dgu.plugin:DrupalAuthPlugin
ckanext.dgu.authentication.drupal_auth:DrupalAuthMiddleware
Middleware to log-in the user based on Drupal cookies and requests to Drupal.
ckanext.dgu.plugin:AuthApiPlugin
ckanext.dgu.authorize
Changes permissions:
- hierarchy structure - edit package/group if is an editor for the group or an admin for the group or its parents.
- All package creations/edits need an API key - no anonymous ones
- UKLP packages can't be edited through the form or API - only by harvesting (unless sysadmin)
- ONS packages can't be edited through the form or API - only by ONS loader (unless sysadmin)
- Packages can only be deleted by sysadmin or UKLP packages by their editor/admin.
- Users can only be viewed by the user and sysadmin
- User list can only be viewed by sysadmin, editors, admins.
ckanext.dgu.plugin:PublisherPlugin
ckanext.dgu.controllers.publisher:PublisherController
Sets 'ckan.auth.profile' to 'publisher' (and same for harvesting: 'ckan.harvest.auth.profile' = 'publisher'). Publisher controller, based on Group:
- Publisher browse page includes hierarchy, alpha-browse and search.
- Apply to be a publisher editor or admin.
- Edit admins/editors for a publisher
- Publisher read shows publisher hierarchy, search pane with results and pager
- Display publisher's admins/editors for appropriate users.
- Report pages - users not assigned to groups, groups without admins, publishers vs users, users When user is created, flashes "You can now apply for publisher access"
ckanext.dgu.plugin:ThemePlugin
ckanext.dgu.controllers.data:DataController
ckanext.dgu.controllers.tag:TagController
ckanext.dgu.controllers.reports:ReportsController
from ckanext.dgu.lib import helpers
ckanext/dgu/theme/templates
ckanext/dgu/theme/public
Data, Tag and Reports Controllers. Templates, helper functions. Random extras:
- Viewing user names (e.g. in History) change them for dept if not editor/admin.
- Add 'Vary: Cookie' header to all responses.
ckanext.dgu.plugin:SearchPlugin
from ckanext.dgu.search_indexing import SearchIndexing
Add fields to search index. Default sort-by. Escape SOLR characters. Search field weighting adjusted.
ckanext.dgu.forms.publisher_form:PublisherForm
New group form, type 'publisher' with schema. Added fields: contact, foi contacts, category, abbreviation.
ckanext.dgu.forms.dataset_form:DatasetForm
ckanext.dgu.forms.validators
ckanext.dgu.schema:GeoCoverageType
New dataset form. Lots of schema and validation customisation.
ckanext.dgu.plugin:ApiPlugin
ckanext.dgu.controllers.api:DguApiController
ckanext.dgu.controllers.api:DguReportsController
ckanext.dgu.lib.reports
Util API for Drupal - latest-datasets (front page), dataset-count (front page), revisions (unused). Reports, starting with organisation_resources.
ckanext.dgu.plugin:ResourceModificationPlugin ckanext.dgu.plugin:ResourceURLModificationPlugin
When a resource is created/deleted/URL-changed, this updates the last_major_modification date of its package.
ckanext/dgu/schema.py - mostly not used now
ckanext/dgu/drupalclient.py - for getting user info from Drupal
dgu/ckanext/dgu/bin/ - scripts used at one time or another
dgu/ckanext/dgu/commands/ - scripts used at one time or another
This is how to install ckanext-dgu, ckan and their dependencies into a python virtual environment::
virtualenv pyenv
pip -E pyenv install -e git+https://github.com:okfn/ckanext-dgu.git#egg=ckanext-dgu
pip -E pyenv install -e git+https://github.com/okfn/ckan.git#egg=ckan
pip -E pyenv install -r pyenv/src/ckan/pip-requirements.txt
pip -E pyenv install -r pyenv/src/ckanext-dgu/pip-requirements.txt
There are plenty of other little details about getting CKAN running under Apache, SOLR config etc in the CKAN README.
When configuring Apache, DGU enables some modules SSL and for ckanext-os. This is the complete list:: sudo a2enmod proxy wsgi headers proxy_http rewrite ssl
Different parts of the DGU extension require options to be set in the CKAN configuration file (.ini) in the [app:main] section
To use the DGU package form specify::
ckan.plugins = dgu_form dgu_package_form
For the Drupal RPC connection (for user data etc.) supply the hostname, and credentials for HTTP Basic Authentication (if necessary)::
dgu.xmlrpc_domain = drupal.libre.gov.fr:80
dgu.xmlrpc_username = ckan
dgu.xmlrpc_password = letmein
The DGU-version of the SOLR schema is required instead of the CKAN SOLR schema. Whether you use a single or mult-core SOLR setup, you'll need a link to the DGU SOLR schema like this::
sudo ln -s /home/okfn/pyenv/src/ckanext-dgu/config/solr/schema-1.4-dgu.xml /etc/solr/conf/schema.xml
Creating the database is as usual for CKAN::
db=dgu
db_user=dgu
sudo -u postgres createuser -S -D -R -P $db_user
sudo -u postgres createdb -O $db_user $db
And because we usually use ckanext-dgu also with ckanext-spatial, then PostGIS needs setting up too::
sudo -u postgres psql -d $db -f /usr/share/postgresql/9.1/contrib/postgis-1.5/postgis.sql -v ON_ERROR_ROLLBACK=on
sudo -u postgres psql -d $db -f /usr/share/postgresql/9.1/contrib/postgis-1.5/spatial_ref_sys.sql
sudo -u postgres psql -d $db -c "ALTER TABLE spatial_ref_sys OWNER TO $db_user;"
sudo -u postgres psql -d $db -c "ALTER TABLE geometry_columns OWNER TO $db_user;"
(For more info on that, see the README for ckanext-spatial.)
There is a front-page added to CKAN which describes the Catalogue APIs. The usual CKAN front-page has been moved to /ckan/ .
e.g.::
http://127.0.0.1:5000/ckan/
There are a number of command-line scripts for processing data. Over the years they have been created in three different ways, but more recently we've been focussing on just creating stand-alone python scripts.
To run a python script (in the ckanext/dgu/bin directory) you activate the environment and then call it with python::
. pyenv/bin/activate
python ckanext/dgu/bin/dataset_change_publisher.py --help
To run a console script (listed in setup.py) you should activate the environment first. For example::
. pyenv/bin/activate
bulk_delete --help
To run a paster command (listed in setup.py) you activate the environment and if your CWD is not ckanext-dgu, specify it as the plugin::
. pyenv/bin/activate
paster --plugin=ckanext-dgu prod --help
The DGU theme uses assets (images, javascript, css) from this repo and the shared assets repo::
https://github.com/datagovuk/shared_dguk_assets
Both repos should be cloned next to each other on developer and server machines. If this is not possible then you need to set the dgu.shared_assets_timestamp_path config option to tell CKAN where the shared assets timestamp file is. e.g.::
dgu.shared_assets_timestamp_path = /vagrant/src/shared_dguk_assets/assets/timestamp
Assets are stored in the repo in 'source' form - the form easiest for developers to edit them in. Before they can be served, Grunt must be run on both repos to create the 'public' versions of these files. This does concatenation, minification, compilation of the less, and recording a timestamp (see Gruntfile.js for details).
Source:
images: ckanext/dgu/theme/src/images
javascript: ckanext/dgu/theme/src/scripts
css (less): ckanext/dgu/theme/src/css
And when you run grunt, you get:
images: ckanext/dgu/theme/public/images
javascript: ckanext/dgu/theme/public/scripts
css: ckanext/dgu/theme/public/css
timestamp: ckanext/dgu/theme/timestamp.py
Read more about Grunt installation and running it: https://github.com/datagovuk/shared_dguk_assets/blob/master/README.md
The shared assets need to be served at /assets. On a deployment server, setup nginx or apache to do this. A developer running under paster will find that the shared assets are served autoimatically, as long as the repo is cloned alongside ckanext-dgu and that this config option is not set: dgu.shared_assets_timestamp_path.
#####Unit and functional tests
To test the DGU extension you need the setup with CKAN (see above) and creation of a configured pyenv/src/ckan/development.ini (see http://docs.ckan.org/en/latest/install-from-source.html ).
To run the tests::
{pyenv}/bin/activate
cd {pyenv}/ckanext-dgu
nosetests --ckan ckanext/dgu/tests/
or run them from another directory by specifying the test.ini::
nosetests --ckan --with-pylons={pyenv}/src/ckanext-dgu/test.ini {pyenv}/src/ckanext-dgu/ckanext/dgu/tests/
You can either run the 'quick and dirty' tests with SQLite or more comprehensively with PostgreSQL. Set --with-pylons to point to the relevant configuration - either test.ini or test-core.ini (both from the ckanext-dgu repo, not the ckan one). For more information, see http://docs.ckan.org/en/latest/install-from-source.html .
#####Browser tests
Selenium is used to test a site is operating to a basic minimum standard, and specific checks on javascript elements.
To run the Selenium tests, TODO
#####Address and Connection errors
socket.error: [Errno 98] Address already in useerror: [Errno 111] Connection refused
These errors usually means a previous run of the tests has not cleaned up the Mock Drupal process. You can verify that::
$ ps a | grep mock_drupal
4748 pts/8 S 0:00 /home/dread/hgroot/pyenv-dgu/bin/python /home/dread/hgroot/pyenv-dgu/bin/paster --plugin=ckanext-dgu mock_drupal run -q
Now kill it before running the tests again::
$ kill 4748
#####Config errors
DrupalXmlRpcSetupError: Drupal XMLRPC not configured.
The missing settings that result in this error are to be found in {pyenv}/src/ckanext-dgu/test-core.ini which is also imported into {pyenv}/src/ckanext-dgu/test.ini, so make sure you are specifying either of these config files in your nosetests --with-pylons parameter.
ckan.lib.search.common.SearchIndexError: HTTP code=400, reason=ERROR: [f752f33380e3eec1379cfb89e0fdded8] multiple values encountered for non multiValued field parent_publishers: [london-borough-of-barnet, local-authorities]
This is due to SOLR using the CKAN SOLR schema, rather than the specific DGU one. Change it using the ln -s command above, followed by stopping and starting SOLR.
DGU is an extension for CKAN: http://ckan.org
This README file is part of the DGU Developer Documentation, stored in the
ckanext-dgu repo at ckanext-dgu/doc.
The Developer Docs can be built using Sphinx <http://sphinx.pocoo.org/>_::
python setup.py build_sphinx