academic / harvester

Harvester

The correct software architecture for OAI harvesters

Objectives

tech:

• Golang

• PostgreSQL (Best Solid Open Source Database)

• ElasticSearch (Best search engine and storage)

• Cloud ready, scalable

• Non-blocking Async jobs.

workers:

• OAI Harvester

• Dublin Core Parser

• Elastic Indexer

endpoint:

• REST Search api

• Workers Triggers

• Cron job api

management:

• Admin UI

Roadmap

1. MVP

   Can do basic harvesting and REST API for search

2. 1.0.0

   Admin UI, Docker, SaaS