MERMAID ingestion API Insomnia collection and workflow documentation
Insomnia is an API client used to work with RESTful APIs, and we'll be using this tool in this repository to ingest data
into the MERMAID platform. This repository includes an Insomina workspace import file
(mermaid_ingest_workspace.json
), which handles authentication and loads API calls for ingesting different Sample Unit
type records from CSV files.
Insominia can be downloaded from https://insomnia.rest/
Environments have been created within the Insomnia workspace for
local
: http://localhost:8888: Used for software development testing
dev
: https://dev.collect.datamermaid.org: Used for Quality Assurance testing
of new features, bug fixes, etc.
prod
: https://collect.datamermaid.org: Live application that everyone uses.
Also, the details needed to authenticate and make calls to the MERMAID API have been set up.
Ingest Fish Belt Collect Records
Ingest Benthic PIT Collect Records
Ingest Bleaching Records
file
: Select file to uploadprotocol
: Depending on theRequest
, this value has been preset tobleachingqc
,benthicpit
orfishbelt
dryrun
:- if set to
true
, records are validated and valid collect records are created but NOT saved to the dastabase. - if set to
false
, records are validated and valid collect records are saved to the database.
- if set to
If you don't have a project to ingest your records into, go to [MERMAID Collect](https://collect.datamermaid.org /) (or MERMAID Collect dev for dev, etc.) to create a new project. Your project must include all Sites, Mangement Regimes and Observers that are used in your CSV data. Sites and mangement regimes' names match the names used in the CSV file, and the emails of the observers match the observer emails used in the CSV file. Below is an illustration of a CSV file and site, management regimes and observer records that have been added to a project using the Collect application.
-
Set environment:
Use Local
,Use Dev
, orUse Prod
-
Change
<PROJECT>
placeholder that can be found in the URL section of Insomina to the id of the project that you will be ingesting records into.
Example:
api_url/projects/<PROJECT>/collectrecords/ingest/
# Change to...
api_url/projects/4080679f-1145-4d13-8afb-c2f694004f97/collectrecords/ingest/
- In the Mutlipart tab, next to the
file
parameter, select CSV file. - In the Mutlipart tab, next to the
dryrun
parameter, settrue
orfalse
. (What??) - Click
Send
If ingest was successful (200 OK), you will see JSON-formatted results in the righthand Insomnia pane, and your records will be available to view in the Collect application.
If ingest was unsuccessful (400 Bad Request), you will recieve a response in the righthand Insomnia pane listing out validation errors and where to find them in the CSV file. Interpreting unsuccessful ingest has more details on where to find your invalid records.
Other ingest failure types
500 Internal Server Error
: There can be many reasons why you get this error but one notable one is using a CSV file that does not follow the correct column naming (see Ingest template files).
Ingested records can now be validated and submitted in the Collect application.
- Navigate to Collect application
- Click on the project with the new ingested records.
- New records are now available to go through the Validate and Submit workflow. Details can be found here
[
{
"Observer emails *": { <---- Column name in CSV file.
"description": [
"dustin@spageo.com doesn't exist" <---- Description of why it is invalid.
]
},
"$row_number": 2 <---- Row in CSV file.
},
{
"Fish name *": {
"description": [
"This field may not be null."
]
},
"$row_number": 12
}
]
Ingest files follow a strict column naming schema. To make life a bit easier, you can find templates for the supported ingest Sample Unit Types in this respository.
- Fish Belt: fishbelt_template.csv
- Benthic PIT: benthic_pit_template.csv
- Bleaching PIT: bleaching_template.csv
NOTE: In each template, column names that end with *
are required.
NOTE: A recent change to Width to support width ranges, ensure you are using the correct CSV is using the correct names