If you are new to this project and problem domain, I recommend reading the companion article (https://bahr.dev/2022/01/06/point-in-time-scheduler/) and the articles that it links to.
This project is the result of me writing 4 previous blog posts, at least 5 other posts on ad-hoc scheduling from the serverless community, and more than 5 years of waiting on AWS to offer a solution for ad-hoc scheduling. They finally released such a solution in November 2022 with the EventBridge Scheduler. See the section "Further Reading" at the end of this Readme. There are also additional repositories from previous experiments at https://github.com/bahrmichael/aws-scheduler and https://github.com/bahrmichael/aws-scheduler-testing.
This is the core of the scheduler project. You can find the frontend at https://github.com/bahrmichael/serverless-scheduler-app and additional documentation at https://github.com/bahrmichael/point-in-time-scheduler.
Please consider the code in this repository as a working prototype for market validation. The authorization with NextAuth, API Keys, and Cognito works, but it needs some work to clean up the code and the various authorization paths.
This project consists of the Infrastructure as Code in the serverless.ts file, various CRUDL APIs and the authorizer in the
/functions/api folder, the heart of the project in /functions/engine and some rudimentary metering in /functions/metering.
The project uses API Gateway's usage plans feature to eventually support metering and billing through AWS marketplace.
Incoming messages are processed by /functions/ingestMessage. If all validations are successful, the message is sent directly into
an SQS queue (if it has less than 10 minutes remaining until release), or into a DynamoDB table for long term storage. SQS supports
message delays of up to 15 minutes.
The engine then picks up messages from the DynamoDB table. This happens in multiple steps to achieve high scalability.
First, /functions/engine/schedulePull loads all registered applications and triggers a run of /functions/pullForOwner for each.
This function name is not up-to-date and should rather be called pullForApplication. It takes an application id, and uses it
to query for messages that can be released within 5 minutes. The messages are then put into the SQS queue from above.
Once the message delay in SQS has expired, the message becomes visible and the Lambda function /functions/releaseMessage sends
them to their destination.
You can find a swagger dashboard at https://app.swaggerhub.com/apis/bahrmichael/Scheduler/0.1.0. Replace the current URL there with https://k5v6qjp3ha.execute-api.us-east-1.amazonaws.com/dev.
I made an attempt at building an SDK, but it turned out a bit more difficult than expected, so it moved down the priority list.
You can still check it out at https://www.npmjs.com/package/point-in-time-scheduler.
There is no demo instance at this time.
Due to the frontend and login features, the initial deployment is a multi-step process.
Run npm i to install the packages. Yarn doesn't work right now due to package conflicts.
To start the deployment, you need to come up with a NEXTAUTH_SECRET. Set it as an environment variable as shown below,
with an AWS profile of your choice (or omit if credentials are present via environment variables):
NEXTAUTH_SECRET=secret AWS_PROFILE=<YOUR-PROFILE> npm run deploy
Wait for this to complete, and note the deployment URL. You will need it again for the frontend deployment as CORE_API_URL.
Once this deployment is complete, you will need to do some click ops to generate an API key for the frontend. All requests are secured through tenant-scoped api keys and usage plans, except for the frontend which currently uses a shared api key and usage plan.
- Go to the API Gateway service, and make sure that the API for the scheduler has been deployed.
- In the setting of the API Gateway service create a new API key and give it a name like "serverless scheduler frontend". Copy the key into a safe place.
- Create a usage plan where you associate the api key and the deployed stage of the api.
Now redeploy the backend service with the api key as the CORE_API_KEY environment variable:
CORE_API_KEY=my-core-api-key NEXTAUTH_SECRET=secret AWS_PROFILE=<YOUR-PROFILE> npm run deploy
You can now go ahead and deploy the frontend. Once you have a Vercel project domain, come back to this repository,
and replace the domain from CallbackURLs: ['https://serverless-scheduler-app-one.vercel.app/api/auth/callback/cognito']
in serverless.ts with your domain.
Deploy the backend service again, and now everything should work nicely together:
CORE_API_KEY=my-core-api-key NEXTAUTH_SECRET=secret AWS_PROFILE=<YOUR-PROFILE> npm run deploy
I decided against tests during the prototyping phase. After that I went straight ahead to canary-like e2e tests, because some code paths are only accessed if there are delays of more than 15 minutes. That felt like too much for a test suite which is part of the deployment process.
- Serverless Scheduler - A research project
- Point In Time Scheduler - This project
- https://medium.com/@zaccharles/there-is-more-than-one-way-to-schedule-a-task-398b4cdc2a75