Jump to: Getting Started | Quick Start | CI/CD | Architecture | Contributing
SLIC Starter :-) is a complete starter project for production-grade serverless applications on AWS. SLIC Starter uses an opinionated, pragmatic approach to structuring, developing and deploying a modern, serverless application with one simple, overarching goal:
Get your serverless application into production fast
- How does SLIC starter help you?
- Application Architecture
- What does it provide?
- Before you Begin!
- Getting Started
- Getting to your First Successful Deployment
- Local Development
- Backend configuration for front end
- Demo
- Code Style and Syntax
- Who is behind it?
- Other Resources
- Troubleshooting
- Contributing
- License
- Serverless development involves a lot of decisions around which approach to take for a multitude of issues. It aims to remove 80% of this decision making and let you focus on building valuable features.
- It is deployable out of the box. Without making any code changes, you have a cloud-hosted production-grade app as a starting point for your product.
- It comes with solutions for the common hard-to-solve problems such as project structuring, deployment, local environments, testing, monitoring and more. Read on to find out more!
- It delivers a fully automated CI/CD build and deployment process that gives reassurances around the state and quality of each environment.
This project is free to use by enterprise, startups, students, educators, enthusiasts and skeptics alike. We actively encourage contributions, suggestions and questions from anyone.
For the CICD pipeline architecture, see CI/CD
SLIC Starter is a complete, working application. By including all the aspects of a real application, SLIC Starter goes beyond a typical demo project. You are encouraged to:
- Take SLIC Starter and copy it
- Use it to learn
- Build your own production application
- Contribute ideas and feedback!
We chose a monorepo approach. Every serverless module (service) is a folder at the parent level of the repo. The monorepo/multirepo decision is always a tricky one but we think monorepo works best here for these reasons.
- Getting started and onboarding developers becomes easier when everything is in one repo.
- Dependencies on common code and libraries are easier when you avoid referencing.
- All applcation code and infrastructure-as-code exist together.
- Changes across multiple services are managed and tracked in the same commits, PRs and merges.
- End-to-end tests exist in the same place as the code under test.
SLIC Starter uses:
- AWS, including Cognito, DynamoDB, Lambda, API Gateway, IAM, Secrets Manager, Systems Manager, S3, Route 53, X-Ray, CodeBuild, CodePipeline, Amplify, SQS, Cloudwatch.
- The Serverless Framework and plugins for serverless modules
- CDK for managing and deploying the CI/CD pipeline
- Node.js for service implementation. This can be replaced with the language of your choice!
Authentication is a difficult problem with constantly-evolving security requirements. SLIC Starter uses Cognito and the Amplify SDK to remove the burden. Cognito can still be complex when getting started. SLIC Starter provides a complete setup with user pool and identity pool, sign-up and login. Amplify is used to authenticate in the web client. There is even a Cognito/Amplify simulation that allows you to work in local development mode, avoiding the need for a real Cognito backend in all cases.
SLIC Starter includes useful, working examples of typical CRUD (create, read, update and delete) actions with a REST API. Currently, these APIs are implemented with DynamoDB and the DocumentClient
API. For an example of this, look at the implementation of the Checklist service.
Upon first deploying SLIC Starter, the API Service will create Record Sets and an API Domain Name for your API. APIGateway and Route53 handles the creation of these resources for us.
A major goal of SLIC Starter is to provide a realistic, working examples for event-driven messaging, including a scalable and real-time message bus.
Following the KISS principle, SLIC Starter only requires the simplest messaging possible for now, so we use CloudWatch Events as inspired by this article.
SLIC Starter has a front-end web application. It uses React, Redux and Material UI. Out of the box, the front end is configured, built, packaged and deployed to S3 and CloudFront with a domain and HTTPS certificate.
- API calls and authentication are managed with AWS Amplify
- The build process automatically looks up the Cognito parameters so you don't have to
- The API domain name is set by convention (The API for
sliclists.com
isapi.sliclists.com
)
Getting continuous integration and deployment (CI/CD) right is one of the most important things in your microservice or serverless project. Having a good foundation here allows you to keep making changes fast. It's also fairly difficult to get right. SLIC Starter has made key choices to help you here.
- SLIC Starter uses multiple AWS accounts for secure isolation of environments. We assume a production, test(staging) and cicd account exist. We have made sure that you can use the same account ID for each of these in case you are restricted to one account and a multi-account setup is not feasible.
- CodePipeline and CodeBuild are used, so the CI/CD process is deployed using Infrastructure-as-Code, just like the serverless application itself. For this, we use the CDK.
- The process dynamically creates a pipeline for each module(service) in the application. An orchestrator pipeline detects which modules need to be built, monitors their pipelines and triggers deployment to the staging account.
- Integration (API) and end-to-end UI tests are run before deployment to production. A manual approval step before deployment to production is included too.
- The entire pipeline is started by a CodeBuild job. CodeBuild is used because it can monitor changes on any branch. This will enable feature branch deployments in the future. This source CodeBuild job also runs a change detection script to determine which services need to be built and deployed. Unchanged services are skipped throughtout the deployment process.
- The CICD stack also include the Pipeline Dashboard application from the Serverless Application Repository, giving you an automatic CloudWatch dashboard of the performance of all pipelines. (Credit to @heitorlessa for the idea and helping to get this working).
SLIC Starter covers automated testing with:
- Unit tests for each service. 100% code coverage is achieved with the exception of the front end!
- Integration (API) tests that exercise the back end
- UI end-to-end tests using TestCafe
All tests can be run in local development mode as well as against a fully-deployed environment. The API and E2E tests are executed against a staging environment before deployment to production.
For details on integration (API) tests, see the README.md in integration-tests
X-Ray is enabled for all services and centralized logging is supported. This is pretty basic in terms of monitoring support so much more is planned and contributions are welcomed.
As a default log centralization solution, SLIC Starter publishes logs to logz.io. The logging module handles this using the logz.io forwarder. This is integrated into each service's Lambda function using the serverless-log-forwarding plugin.
For further details, see the logging README
We use AWS Secrets Manager for storing the GitHub personal access token and AWS Systems Manager Parameter Store for storing other secrets, such as API access tokens.
Coming soon. SLIC Starter will include support for roles and Role-Based Access Control (RBAC).
SLIC Starter is designed to get you up in running with a real-world application as quickly as possible. The fact that we go beyond the average sample application, there is a bit more involved in getting to production. For example:
- We assume that you want to keep the CICD, staging and production accounts separate in most cases. These can be set up under one root account using AWS Organizations. SLIC Starter also now supports single account deployment, so CICD, staging and production environments can be configured to live under the one account.
- SLIC Starter assumes you are using a registered domain (like
sliclists.com
) and will set up DNS entries for use in production (likeapi.sliclists.com
) and staging (stg.sliclists.com
,api.stg.sliclists.com
). If you want to get up and running with a system quickly, you can skip domains by following the QUICK_START.md guide. - By using domains, you will have to take some steps to set up DNS records and allow these to propagate.
- When your application is automatically deploying as part of the CICD process and HTTPS certificates are being created, you (the domain owner) will be sent an email by Amazon Route53 to verify that you are the domain owner.
- You will also have to validate your domains with SES in order to have permissions for emails to be sent.
This section covers a full deployment using multiple accounts with domain names and HTTPS certificates. It takes quite a bit longer than your average sample app since there is DNS and certificate approval involved. If you want to try SLIC Starter out in a single account without domains, go to the QUICK_START.md]
To set up deployment to your own accounts, first run through these steps.
- Decide when DNS name you will use for your application. If you need to register one, the best place to do this is probably in your production account using Amazon Route 53.
- Copy
slic-config.json.sample
toslic-config.json
and edit it to include the AWS Account IDs of your staging, production and CI/CD accounts. For single account deployment the Account ID's of CICD, Production and Staging would all be the same. This file is.gitignore
'd so your account IDs are not commited to Git. Also specify your GitHub repository details and DNS domain for your deployment. Use a domain you own so you can update DNS entries to point to your deployed environment. When the deployment process runs, the domain owner will be sent an email to verify ownership before the deployment completes. - Fork the repository into your own account or organization on GitHub. If you don't use GitHub, you will have to tweak the source project in the CICD module (source-project.ts)
- Enable CodeBuild to access your GitHub repo. The most reliable way to do this is to use a Personal Access Token with access to create webhooks on your repository. The guide from AWS on how to do this is here.
- (Optional - this will be required for repo tagging). Set up GitHub authentication for your repo. Create a GitHub Personal Access Token and add it as an secret with the name
GitHubPersonalAccessToken
in Secrets Manager in the CICD account. See this post for more detail on this approach. - Under the parent directory, open the file
common-config.json
. If desired, you can configure the default regions and domain prefixes by account. If you have no reason to change them, leave the default values as they are. - Create a Mailosaur account. Take the server ID and API key and add them in your CICD account to the Parameter Store as
SecretString
values with the following names
test/mailosaur/serverId
test/mailosaur/apiKey
These are picked up by the integration and end-to-end test CodeBuild projects.
- Create a secret string in System Manager Parameter store with a value used to sign and verify verification codes - the parameter name should be
/STAGE/sharing-service/code-secret
where STAGE is the stage you are deploying to (dev, stg or prod). - Give permissions for your CICD account to deploy to staging and production accounts.
npm install -g serverless
cd cicd/cross-account
AWS_PROFILE=your-staging-account serverless deploy
AWS_PROFILE=your-production-account serverless deploy
- Deploy the CI/CD pipeline to your CICD account.
cd cicd
npm install
npm run build
AWS_PROFILE=your-cicd-account npm run cdk -- bootstrap
SLIC_NS_DOMAIN=your_domain AWS_PROFILE=your-cicd-account npm run deploy
- Trigger your pipeline by commiting your changes to the repository
- Monitor your deployment by viewing the orchestrator pipeline in the AWS Console CodePipeline page.
- Wait for your deployment to fail! Wait, what? Yes, your first deployment will fail. This is expected and all part of the process. Read on to find out more!
The CICD process attempts to build and deploy each service in parallel. This is done so you get quick feedback and to improve the overall deployment speed. It also means that deployment can fail if there are dependencies between services. Out of the box, SLIC Starter has a certs
module that sets up a Route 53 Hosted Zone and some certificates. These are required by the frontend
and checklist-service
services, so those builds will fail if the cerificates aren't ready yet. This is just one example. There are other services that depend on common resources so it will require a few retries in both staging and production before everything is deployed.
Note that deployment of some services can take quite some time! In particular, frontend
deployment will wait until the CloudFront distribution has been created. This can take at least 15 minutes.
You can inspect the failures in the Orchestrator Pipeline view in the CodePipeline console of your CICD account. You can retry the stgDeploy
phase by clicking the Retry button in the pipeline.
Once you get all services in staging successfully deployed, you might find that the test stage fails. This is likely to do with the front end being inaccessible. As we already mentioned, your DNS entries will need to be set up. Let's understand how this all works better!
- CICD will create
NS
andA
records for staging in your staging account. When production deployment happens, it will also createNS
andA
records in your production account. - You need to decide who will own the records for your APEX domain, i.e. the parent domain (such as
sliclists.com
) - If you let your production AWS account own these records, you need to copy the
NS
records from your production Route53 into your domain registration name server settings. You will then need to copy theNS
record (only theNS
record!) for staging into your production Route 53 for the apex domain and give the NS record the staging domain (e.g.,stg.sliclists.com
). - Alternatively, if you want another provider to own the nameserver records for your domain, you should copy all generated DNS records (except
NS
type records) from your staging and production account into that provider's DNS records.
Once you have set up the required DNS configuration and it has propagated, your front end staging application should be available at https://stg.YOUR-DOMAIN.TLD
and you can click Retry on the test phase of the pipeline!
For the email-service
to send emails, you must choose a 'From:' address and set it in parameter store. The paramter name is /<STAGE>/email-service/from-address
. <STAGE>
should be replaced with stg
or prod
as appropriate.
order for the email service to send requests to SES, either the email address or the domain needs to be verified. This is not automatically done as part of the SLIC Starter deployment.
If you are using SLIC with a domain configuration, the verification process is quite straightforward because we use Route 53 for our DNS records. See here for documentation on how to achieve domain verification throught the AWS Management Console.
By default, SES will require validation of each email address to which emails are being sent. To avoid this, you can request a sending limit increase, which will remove your account/region from the SES Sandbox.
Work on a more seamless local development environment is in progress. In the meantime, here's an introduction to running the checklist-service
locally:
In backend services:
sls dynamodb install
Ensure that your AWS credentials are set to some reasonable values. For local development, any dummy value for AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
is sufficient. You can also opt to use an AWS profile if that's your preferred method for specifying AWS credentials.
SLIC_STAGE=local sls offline start --migrate true
When working in local development, the backend configuration is pulled from .env.local. When building the production frontend for any deployed stage, npm run build
will, by default, generate a .env.production
file. This file is .gitignore
d so it will not be committed. The values for this file are dynamically generated using the CloudFormation outputs retrieved from the stage specified using the SLIC_STAGE
environment variable.
SLIC Starter provides a fully-featured application for managing checklists called SLIC Lists. SLIC Starter is self hosting, so SLIC Lists is continuously deployed to sliclists.com from this repository!
SLIC Starter uses Prettier for code formatting and ESLint for syntax checking.
- Prettier is configured to format the code on commit (run manually with
npm run format
in each project) - Linting can be run with
npm run lint
in each project
SLIC Starter is open source and contributions are welcome from everyone. It was started by the team at fourTheorem, also the authors of the book, AI as a Service, a Manning publication on Serverless, AI-enabled applications.
There are many other amazing resources to help you get started, learn and evolve your Serverless practice. Here are a few recommendations!
- Serverless Stack Tutorial - A really well-crafted, step-by-step tutorial covering many best practices
- AWS Serverless Airline Booking - A complete web application built on Serverless AWS with Amplify and GraphQL. The Twitch videos covering the architecture and build of this project are available here on YouTube
- I get this error in the CodeBuild source project:
87/101 | 1:05:21 PM | CREATE_FAILED | AWS::CodeBuild::Project |
sourceProject (sourceProjectBCA86C81) Failed to call CreateWebhook, reason: Repository not found or permission denied.
- This is because your CodeBuild configuration does not have access to your GitHub account. To grant access, create a CodeBuild project manually in the AWS Console and grant access to your repository. This project can be safely deleted afterwards once CodeBuild has been granted access.
See CONTRIBUTING.md
Copyright fourTheorem Ltd. 2018-2019. Distributed under the MIT License. See LICENCE