Enduro is a tool designed to automate the processing of transfers in multiple Archivematica pipelines. It's part of a preservation solution that is being used by the National Health Archive (NHA) and the National Center for Truth and Reconciliation (NCTR).
It's a proof of concept at its very early stages. It aims to cover our client's needs while exploring new and innovative ways to build durable and fault-tolerant workflows suited for preservation.
Visit https://enduroproject.netlify.com for more details.
Enduro uses Tilt to set up a local environment building the Docker images in a Kubernetes cluster. It has been tested with k3d, Minikube and Kind.
A local Kubernetes cluster:
It can run with other solutions like Microk8s or Docker for Desktop/Mac and even against remote clusters, check Tilt's Choosing a Local Dev Cluster and Install documentation for more information to install these requirements.
Additionally, follow the Manage Docker as a non-root user post-install guide
so that you don’t have to run Tilt with sudo.
Start a local Kubernetes cluster with a local registry. For example, with k3d:
k3d cluster create sdps-local --registry-create sdps-registry
Make sure kubectl is available and configured to use that cluster:
kubectl config view
Clone this repository and move into its folder:
git clone https://github.com/artefactual-sdps/enduro.git
cd enduro
Bring up the environment:
tilt up
While the Docker images are built/downloaded and the Kubernetes resources are
created, hit space to open the Tilt UI in your browser. Check the Tilt UI
documentation to know more about it.
There are four services available from the host:
- Enduro dashboard: http://localhost:3000
- Minio console: http://localhost:7460 (username: minio, password: minio123)
- Temporal UI: http://localhost:7440
- Opensearch dashboards: http://localhost:7500
Tilt will watch for file changes in the project folder and it will sync those
changes, rebuild the Docker images and recreate the resources when necessary.
The enduro-dashboard uses Vite to serve the application in development with
hot reload. The enduro and enduro-a3m-worker services require rebuilding
the entire images - these will take longer to update.
Additionally, there are refresh buttons on each resource in the Tilt UI that allow triggering manual updates and re-executing jobs and local resources.
Run ctrl-c on the terminal where tilt up is running and stop the cluster
with:
k3d cluster stop sdps-local
To start the environment again:
k3d cluster start sdps-local
tilt up
To remove the resources created by Tilt in the cluster, execute:
tilt down
However, that won't remove all the persistent volumes at the moment, to do so run:
kubectl delete pvc,pv --all
Deleting the cluster will remove all the resources from above, but it will also remove the container registry with the Docker images and the cluster container from the host. With k3d, run:
k3d cluster delete sdps-local
In the Tilt UI header there is a cloud icon/button that allows you to configure
and trigger an upload to the sips bucket in Minio. Click the caret to set the
path to a file/directory in the host and a Minio object name, then click the
cloud icon to trigger the upload.
For example, to upload an existing sample ZIP from the project folder (make
sure you update /path/to/enduro to the proper project folder in the host):
- Host path:
/path/to/enduro/hack/sampledata/StructB-AM.zip - Object name:
StructB-AM.zip
The setup of the Minio buckets and the communication between Minio and Redis
is sometimes not setup properly. To solve it, from the Tilt UI, restart the
minio resource and then trigger the minio-setup-buckets resource.