Animl API

An AWS Lambda-based, GraphQl interface for performing CRUD operations on camera trap data stored in MongoDB.

`Related repos`

Animl frontend http://github.com/tnc-ca-geo/animl-frontend
Animl base program http://github.com/tnc-ca-geo/animl-base
Animl ingest function http://github.com/tnc-ca-geo/animl-ingest
Animl ML resources http://github.com/tnc-ca-geo/animl-ml
Animl desktop app https://github.com/tnc-ca-geo/animl-desktop

`Overview`

Animl is an open, extensible, cloud-based platform for managing camera trap data. We are developing this platform because there currently are no software tools that allow organizations using camera traps to:

ingest data from a variety of camera trap types (wireless, SD card based, IP, etc.)
systematically store and manage images in a single centralized, cloud-based repository
upload custom object detection and species clasification ML models and configure automated assisted-labeling pipelines
Offer frontend web application to view images, review ML-assisted labels, perform manual labeling
Offer an API for advanced querying and analysis of camera trap data
Offer tools for exporting ML model training data

This repository contains an AWS Lambda-based, GraphQl API for storing and fetching cameratrap data from a MongoDB database. The stack and and it's associated deployment resources are managed with the Serverless Framework. The stack includes:

Apollo Server
MongoDB Atlas via Mongoose

`Development`

Prerequisits

The instructions below assume you have the following tools globally installed:

Node & npm
Serverless
aws-cli

Create "animl" AWS config profile

The name of the profile must be "animl" (because it's referenced in the serverless.yml file). Good instructions here.

Make a project direcory, clone this repo, and install dependencies

mkdir animl-api
cd animl-api
git clone https://github.com/tnc-ca-geo/animl-api.git
cd animl-api
npm install

Preparing remote config variables

The API depends on remote secrets and parameters that are stored in AWS Secrets Manager and AWS Systems Manager Parameter Store, respectively. Most of the params are generated by this project's serverless config file and the config files of other services upon which this app depends, but some must be created manually via the AWS console. To make sure you have the correct secrets and parameters available, do the following:

Make sure you've deplpyed animl-ingest, animl-frontent, and mira-api in the same staging env (dev/prod) as the environtment you intend to deploy animl-api.
We currently depend on a CloudFormation template ProductOps created called UserPool that creates and manages all of the resources related to Auth/Auth. This is not tracked in version control (but it probably should be), as it's critical and is responssible for generating SSM Params upon which this app depends. Make sure that that stack has been created.
Two important SSM Params, /ml/megadetector-api-key-[env] and /db/mongo-db-url-[env] contain secret keys so need to be created manually in the AWS console. Be sure to create versions for all envs you plan on deploying.

Seeding db

You'll need to create the DB in MongoDB Atlas, but once you have, a script for seeding the DB with default records can be found at animl-api/src/scripts/seedDB.js. If the DB hasn't been seeded yet, you can do so by running the following command from the root directory:

npm run seed-db-dev 
# or, do seed the production db:
npm run seed-db-prod

Local testing and deployment

To test the Lambda locally with serverless-offline, run:

npm run start

To deploy the Cloudformation development stack, run:

npm run deploy-dev

To deploy the Cloudformation production stack, run:

npm run deploy-prod

`Data managment`

There are a handful of scripts in the src/scripts/ directory to assist with managing data in both the production and dev databases.

`Creating backups`

To create a complete JSON export of all collections in a DB, run:

npm run export-db-dev   // export dev db
npm run export-db-prod  // export prod db

`Importing data from a backups`

TODO: write and test importDb.js

`Updating documents in MongoDB`

updateDocuments.js is a working template for writing targeted data updates. It can be adapted to perform specific deletions/updates. You can run it with the following:

npm run update-docs-dev   // update dev db
npm run update-docs-prod  // update prod db

tnc-ca-geo / animl-api