This Lambda is a simple service that handles the data flow of the ResearchNow ingest pipeline. It reads records fed to it via multiple Kinesis streams and passes the resulting records to the proper step in the metadata enhancement/parsing pipeline. When complete it will pass the completed metadata block to the database for storage.
- Clone repository and run
npm install - Copy .env.sample to .env and adjust necessary values (recommended settings are noted)
- install node-lambda globally with
npm install -g node-lambdaif it does not exist - Copy the local/development/production.env.sample and set the appropriate values
- Copy the event_sources_*.env.sample files and set for the appropriate values
- Run the appropriate commands in
package.jsonnpm run local-runwill run the Lambda with values provided in an event.json filenpm run deploy-*will package and deploy the Lambda to the designated environment
This uses a stage variable to designate the step of the enhancement process where to pass the current record:
- NONE/new: Pass to the first stage in the metadata process (currently MetadataWrangler)
- mw: Processed by MetadataWrangler, pass to the OCLC enhancer
- oclc: Processed by MW and OCLC, mark as complete
- complete: Finished, pass to database manager
- Add additional Enhancement steps (LC, Getty, etc)
- Add Kinesis/DLQ for failed records for re-processing