GitHub Self-Hosted Runners CDK Constructs

Use this CDK construct to create ephemeral self-hosted GitHub runners on-demand inside your AWS account.

Easy to configure GitHub integration with a web-based interface
Customizable runners with decent defaults
Multiple runner configurations controlled by labels
Everything fully hosted in your account
Automatically updated build environment with latest runner version

Self-hosted runners in AWS are useful when:

You need easy access to internal resources in your actions
You want to pre-install some software for your actions
You want to provide some basic AWS API access (but aws-actions/configure-aws-credentials has more security controls)
You are using GitHub Enterprise Server

Ephemeral (or on-demand) runners are the recommended way by GitHub for auto-scaling, and they make sure all jobs run with a clean image. Runners are started on-demand. You don't pay unless a job is running.

API

The best way to browse API documentation is on Constructs Hub. It is available in all supported programming languages.

Providers

A runner provider creates compute resources on-demand and uses actions/runner to start a runner.

	EC2	CodeBuild	Fargate	ECS	Lambda
Time limit	Unlimited	8 hours	Unlimited	Unlimited	15 minutes
vCPUs	Unlimited	2, 4, 8, or 72	0.25 to 4	Unlimited	1 to 6
RAM	Unlimited	3gb, 7gb, 15gb, or 145gb	512mb to 30gb	Unlimited	128mb to 10gb
Storage	Unlimited	50gb to 824gb	20gb to 200gb	Unlimited	Up to 10gb
Architecture	x86_64, ARM64	x86_64, ARM64	x86_64, ARM64	x86_64, ARM64	x86_64, ARM64
sudo	✔	✔	✔	✔	❌
Docker	✔	✔ (Linux only)	❌	✔	❌
Spot pricing	✔	❌	✔	✔	❌
OS	Linux, Windows	Linux, Windows	Linux, Windows	Linux, Windows	Linux

The best provider to use mostly depends on your current infrastructure. When in doubt, CodeBuild is always a good choice. Execution history and logs are easy to view, and it has no restrictive limits unless you need to run for more than 8 hours.

EC2 is useful when you want runners to have complete access to the host
ECS is useful when you want to control the infrastructure, like leaving the runner host running for faster startups
Lambda is useful for short jobs that can work within time, size and readonly system constraints

You can also create your own provider by implementing IRunnerProvider.

Installation

Install and use the appropriate package

Python

Install

Available on PyPI.

pip install cloudsnorkel.cdk-github-runners

Use

from cloudsnorkel.cdk_github_runners import GitHubRunners

GitHubRunners(self, "runners")

TypeScript or JavaScript

Install

Available on npm.

npm i @cloudsnorkel/cdk-github-runners

Use

import { GitHubRunners } from '@cloudsnorkel/cdk-github-runners';

new GitHubRunners(this, "runners");

Java

Install

Available on Maven.

<dependency>
   <groupId>com.cloudsnorkel</groupId>
   <artifactId>cdk.github.runners</artifactId>
</dependency>

Use

import com.cloudsnorkel.cdk.github.runners.GitHubRunners;

GitHubRunners.Builder.create(this, "runners").build();

Install

Available on GitHub.

go get github.com/CloudSnorkel/cdk-github-runners-go/cloudsnorkelcdkgithubrunners

Use

import "github.com/CloudSnorkel/cdk-github-runners-go/cloudsnorkelcdkgithubrunners"

NewGitHubRunners(this, jsii.String("runners"))

.NET

Install

Available on Nuget.

dotnet add package CloudSnorkel.Cdk.Github.Runners

Use

using CloudSnorkel;

new GitHubRunners(this, "runners");

Use GitHubRunners construct in your code (starting with default arguments is fine)
Deploy your stack

Look for the status command output similar to aws --region us-east-1 lambda invoke --function-name status-XYZ123 status.json

 ✅  github-runners-test

✨  Deployment time: 260.01s

Outputs:
github-runners-test.runnersstatuscommand4A30F0F5 = aws --region us-east-1 lambda invoke --function-name github-runners-test-runnersstatus1A5771C0-mvttg8oPQnQS status.json

Execute the status command (you may need to specify --profile too) and open the resulting status.json file
Open the URL in github.setup.url from status.json or manually setup GitHub integration as an app or with personal access token
Run status command again to confirm github.auth.status and github.webhook.status are OK
Trigger a GitHub action that has a self-hosted label with runs-on: [self-hosted, linux, codebuild] or similar
If the action is not successful, see troubleshooting

Customizing

The default providers configured by GitHubRunners are useful for testing but probably not too much for actual production work. They run in the default VPC or no VPC and have no added IAM permissions. You would usually want to configure the providers yourself.

For example:

let vpc: ec2.Vpc;
let runnerSg: ec2.SecurityGroup;
let dbSg: ec2.SecurityGroup;
let bucket: s3.Bucket;

// create a custom CodeBuild provider
const myProvider = new CodeBuildRunnerProvider(this, 'codebuild runner', {
   labels: ['my-codebuild'],
   vpc: vpc,
   securityGroups: [runnerSg],
});
// grant some permissions to the provider
bucket.grantReadWrite(myProvider);
dbSg.connections.allowFrom(runnerSg, ec2.Port.tcp(3306), 'allow runners to connect to MySQL database');

// create the runner infrastructure
new GitHubRunners(this, 'runners', {
   providers: [myProvider],
});

Another way to customize runners is by modifying the image used to spin them up. The image contains the runner, any required dependencies, and integration code with the provider. You may choose to customize this image by adding more packages, for example.

const myBuilder = FargateRunnerProvider.imageBuilder(this, 'image builder');
myBuilder.addComponent(
  RunnerImageComponent.custom({ commands: ['apt install -y nginx xz-utils'] }),
);

const myProvider = new FargateRunnerProvider(this, 'fargate runner', {
   labels: ['customized-fargate'],
   imageBuilder: myBuilder,
});

// create the runner infrastructure
new GitHubRunners(this, 'runners', {
   providers: [myProvider],
});

Your workflow will then look like:

name: self-hosted example
on: push
jobs:
  self-hosted:
    runs-on: [self-hosted, customized-fargate]
    steps:
      - run: echo hello world

Windows images can also be customized the same way.

const myWindowsBuilder = FargateRunnerProvider.imageBuilder(this, 'Windows image builder', {
   architecture: Architecture.X86_64,
   os: Os.WINDOWS,
});
myWindowsBuilder.addComponent(
   RunnerImageComponent.custom({
     name: 'Ninja',
     commands: [
       'Invoke-WebRequest -UseBasicParsing -Uri "https://github.com/ninja-build/ninja/releases/download/v1.11.1/ninja-win.zip" -OutFile ninja.zip',
       'Expand-Archive ninja.zip -DestinationPath C:\\actions',
       'del ninja.zip',
     ],
   }),
);

const myProvider = new FargateRunnerProvider(this, 'fargate runner', {
   labels: ['customized-windows-fargate'],
   imageBuilder: myWindowsBuilder,
});

new GitHubRunners(this, 'runners', {
   providers: [myProvider],
});

The runner OS and architecture is determined by the image it is set to use. For example, to create a Fargate runner provider for ARM64 set the architecture property for the image builder to Architecture.ARM64 in the image builder properties.

new GitHubRunners(this, 'runners', {
   providers: [
      new FargateRunnerProvider(this, 'fargate runner', {
         labels: ['arm64', 'fargate'],
         imageBuilder: FargateRunnerProvider.imageBuilder(this, 'image builder', {
            architecture: Architecture.ARM64,
            os: Os.LINUX_UBUNTU,
         }),
      }),
   ],
});

Architecture

Troubleshooting

Runners are started in response to a webhook coming in from GitHub. If there are any issues starting the runner like missing capacity or transient API issues, the provider will keep retrying for 24 hours. Configuration issue related errors like pointing to a missing AMI will not be retried. GitHub itself will cancel the job if it can't find a runner for 24 hours. If your jobs don't start, follow the steps below to examine all parts of this workflow.

Always start with the status function, make sure no errors are reported, and confirm all status codes are OK
Make sure runs-on in the workflow matches the expected labels set in the runner provider
Diagnose relevant executions of the orchestrator step function by visiting the URL in troubleshooting.stepFunctionUrl from status.json
1. If the execution failed, check your runner provider configuration for errors
2. If the execution is still running for a long time, check the execution events to see why runner starting is being retried
3. If there are no relevant executions, move to the next step
Confirm the webhook Lambda was called by visiting the URL in troubleshooting.webhookHandlerUrl from status.json
1. If it's not called or logs errors, confirm the webhook settings on the GitHub side
2. If you see too many errors, make sure you're only sending workflow_job events
When using GitHub app, make sure there are active installations in github.auth.app.installations

All logs are saved in CloudWatch.

Log group names can be found in status.json for each provider, image builder, and other parts of the system
Some useful Logs Insights queries can be enabled with GitHubRunners.createLogsInsightsQueries()

To get status.json, check out the CloudFormation stack output for a command that generates it. The command looks like:

aws --region us-east-1 lambda invoke --function-name status-XYZ123 status.json

Monitoring

There are two important ways to monitor your runners:

Make sure runners don't fail to start. When that happens, jobs may sit and wait. Use GitHubRunners.metricFailed() to get a metric for the number of failed runner starts. You should use this metric to trigger an alarm.
Make sure runner images don't fail to build. Failed runner image builds mean you will get stuck with out-of-date software on your runners. It may lead to security vulnerabilities, or it may lead to slower runner start-ups as the runner software itself needs to be updated. Use GitHubRunners.failedImageBuildsTopic() to get SNS topic that gets notified of failed runner image builds. You should subscribe to this topic.

Other useful metrics to track:

Use GitHubRunners.metricJobCompleted() to get a metric for the number of completed jobs broken down by labels and job success.
Use GitHubRunners.metricTime() to get a metric for the total time a runner is running. This includes the overhead of starting the runner.

Other Options

philips-labs/terraform-aws-github-runner if you're using Terraform
actions/actions-runner-controller if you're using Kubernetes

CloudSnorkel / cdk-github-runners

GitHub Self-Hosted Runners CDK Constructs

API

Providers

Installation

Install

Use

Install

Use

Install

Use

Install

Use

Install

Use

Customizing

Architecture

Troubleshooting

Monitoring

Other Options

About

Languages