Legacy Challenge Migration CLI Tool
Migration script used to migrate Challenges, Resources, and Resource Roles from Informix to DynamoDB. It runs on a scheduled basis and also on-demand by exposing an API allowing admins to manually trigger the migration.
Development deployment status
Production deployment status
Intended use
- Data migration script to move v4/Informix to v5 Dynamo/ES
Related repos
Prerequisites
- NodeJS
- Elasticsearch(v6.3.1)
- DynamoDB
- Informix
- Docker(CE 17+)
- Docker Compose
Configuration
See config/default.js
. Most of them is self explain there.
PORT
: API server port; default to3001
API_VERSION
: API version; default tov5
SCHEDULE_INTERVAL
: the interval of schedule; default to5
(minutes)CHALLENGE_TYPE_API_URL
Challenge v4 api url from which challenge types data are fetched.CHALLENGE_TIMELINE_API_URL
Challenge v5 api url from which challenge timelines are fetched.CREATED_DATE_BEGIN
A filter; if set, only records in informix created after the date are migrated.BATCH_SIZE
Maximum legacy will be load at 1 queryERROR_LOG_FILENAME
Filename for data that error to migrate.RESOURCE_ROLE
List of resource role to be included in migration
Local Deployment
Foreman Setup
To install foreman follow this link To know how to use foreman follow this link
If you run something with nf run npm run <something>
instead of npm run <something>
, it'll load in a .env
file from the root of the project
Deployment
To simplify deployment, we're using docker. To build the images or run the container:
cd <legacy-challenge-migration-cli>/docker
docker-compose up
This will automatically build the image if have not done this before. After container has been run, go to container shell and install dependencies:
docker exec -ti legacy-challenge-migration-cli bash
npm i
Command for API
- Inside the docker container, start the express server:
npm start
Migrations
For local migrations
There are two parts need to be updated for local migrations,
./src/models/challenge.js
throughput: 'ON_DEMAND',
should be updated tothroughput:{ read: 4, write: 2 },
./config/default.js
Two aws config should be uncommented
and env variable IS_LOCAL_DB
should be set to true before you continue to the next steps.
Run a migration
- To run a migration, the command should be:
MIGRATION=<name-of-migration-file> nf run npm run migrate
Example:MIGRATION=001-migrate-taskData nf run npm run migrate
Production deployment
- TBD
Running tests
- TBD
Running tests in CI
- TBD