recalcitrantsupplant / spatial-data

IDN Spatial Data

This repository contains source data, data generation scripts and final data for the Indigenous Data Network (IDN)'s set of demonstration spatial datasets.

These datasets are accessed live, online, via the IDN’s Demonstration Catalogue. See https://idnau.org for access.

Datasets

• Australian Government Indigenous Programs & Policy Locations (AGIL)

• Native Title Databases at the National Native Title Tribunal

Data Folders

Each dataset in this repository is stored in a folder within the top-level folder datasets/, for example the "Australian Government Indigenous Programs & Policy Locations" (AGIL) dataset" is stored in datasets/agil/.

Each dataset folder has several folders and a file:

• data/ - final RDF data for this dataset.

  • this is all you need if you want to get this dataset’s content

  • multiple files where data exceeds 100MB

• scripts/ - Python and perhaps other scripts to produce the final data

• source/ - source data for the dataset, if we have it and are allowed to share it

  • could be Excel files, SHAPE files etc.

• {DATASET-ID}-metadata.ttl - an RDF file containing some basic Dataset and at least one Feature Collection’s metadata

• to be combined with the dataset’s data by the scripts

• included in the final RDF data thus not needed for data use

The top-level folder background-rdf/ contains ontologies that should be loaded with the spatial data to give labels to properties.

data/

These final data for these datasets conform to the with Geoscience Australia’s Digital Atlas of Australia (DAA)'s data model. This means these datasets can be used interchangeable with them and are known to display well in with the Prez Linked Data tool.

scripts/

The conversion scripts here are all quite similar. They use Python scripts which:

• read the source data

  • using appropriate adaptors from public Python packages

• extract information relevant to the DAA data model

  • see the FSDF Backbone Model

• formulate RDF data

  • using Python’s RDFLib library

  • as an in-memory graph

• save the data to a file

  • combining it with static metadata for the dataset stored in an RDF file

• validate the data

  • according to the FSDF Backbone Model’s compounded validator

Each dataset’s source data, scripts, static metadata and results are stored per-dataset in folders within datasets/ in this repository.

script usage

This repository’s scripts are Python scripts that are compatible with Python 3.10 and use the Poetry python package manager.

To run these scripts, you will need to:

1. clone this repository to your local computer

2. establish a Python environment with all the dependent packages

   1. the Poetry documentation describes this

3. execute the scripts

   1. you can probably just run ~$ python {DATASET-ID}-etl.py in the scripts/ directory for aech dataset

4. check the data/ folder in each dataset’s folder

   1. for {DATASET-ID}.ttl - the resultant data file

License & Rights

The contents of this repository is licensed under Creative Commons 4.0 International. See the LICENSE file in the repository for details.

All content is © Indigenous Data Network, 2022

Contacts

Initial technical development of this work is being undertaken by the following, for the Indigenous Data Network:

Sandra Silcot
Data Custodian
University of Melbourne
Indigenous Data Network

Nicholas J. Car
Data Architect
Kurrawong AI
[e-mail]