This repository contains different components needed to produce and maintain Hakai ERDDAP servers and associated datasets. Server update status:
All datasets made available within the datasets.d
folder in the ERDDAP XML format are made available on the production server.
Hakai deploy ERDDAP as docker containers by using the docker-erddap image. Continuous Integration is handled via the erddap-deploy actions and the container configuration is handled via CapRover applications.
See GitHub Deployments for all active deployments maintained via this repository.
ERDDAP is deployed on <catalogue.hakai.org>, via docker-compose.
See local .env
file for configuration.
For local development, make a copy of sample.env
file as .env
.
Update the environment variables to match the deployed parameters.
Omit the email parameters and baseHttpsUrl and baseUrl.
Add test files if needed within the datasets/
directory.
Run docker-compose
docker-compose up -d
If successful, you should be able to access your local ERDDAP instance at http://localhost:{HOST_PORT}/erddap (default: http://localhost:8080/erddap)
This is a step by step procedure to generate a new dataset:
- See development for create a local testing environment.
- If the new dataset is form files, add the datawithin the datasets folder. Create a subdirectory for this dataset and structure the data accorindgly.
- Active your docker daemon
- Run the bash command:
sh Dasds.sh
- Answer the different questions accordingly to generate a first draft of the xml associated with this dataset.
- Importat! All the paths requested by ERDDAP should be referencing the base
directory (/datasets) within the container which is the local
./datasets/
folder mounted within the container. - Once all the answers completed, see what ERDDAP returned to you.
- If sucessfully, completed ERDDAP should return the dataset.xml associated to this dataset.
- You can also retrieve it at: here within the log folder
- Copy this xml file to ./datasets.d, and name the file with its datasetID.
- Add the different missing metadata.
- Once satisfied, you can test the dataset with the command
sh DasDds.sh
- Fix all the issues brought up by DasDds.sh
- Once completed, restart your local instance of ERDDAP and see if that dataset is now available.
- Once satisfied, commit the changes to the
development
branch. - If needed, copy the file data to <goose.hakai.org> server within (/data2/erddap_data) and match the path you've used initially within ./datasets/
- Any changes to the development branch should be reflected on the development erddap autimacally via a github action
- Review development version.
- From the development branch, create a release branch.
- Merge Master to the release branch, ignore any changes associated to other datasets in development
- Create PR to master
- Revise PR
- Make sure that the file data (if needed) is available within the
catalogue.hakai.org
server at/data/erddap_data
- Once happy with the PR, merge it.
Hakai-erddap
is using the hakai-metadata-conversion
package to sync
periodically the different datasets metadata based on the latest changes
made within the Hakai Catalogue.
If any changes are available a PR to development should be automatically generated. Merge the changes to development, and follow a similar method as described within the production server section.
All views and tables generated from the different SQL queries made available in
the view
directory are run nightly from the hecate.hakai.org server from the
master branch with the bash script erddap_create_views.sh
All commits to this repository are tested by different linter through a PR or commit to the development and master branches:
- python: black, isort
- sql: sqlfluff
- markdown
- xml
We are using the super-linter library to generate the different automated integration tests.
If the linter tests and erddap_deploy tests pass, changes will automatically be reflected on the associated deployment via:
- Redeployment of the application: on changes to erddap/**/*,Dockerfile or environment variables.
- Update of the datasets.xml: on changes to datasets.d or datasets.xml