This guide will walk you through the process of implementing automatic scaling with GitHub Actions and self-hosted runners on your existing Amazon EKS cluster. This setup will allow you to leverage your own infrastructure efficiently and adapt to varying workloads while utilizing the benefits of GitHub Actions.
- An existing Amazon EKS cluster.
- EC2 instances or virtual machines that will act as self-hosted runners within your EKS cluster.
- Basic knowledge of GitHub Actions and how to set up self-hosted runners.
-
Set up self-hosted runners
- Launch EC2 instances or provision virtual machines that will serve as your self-hosted runners within the EKS cluster. Ensure these instances have the necessary resources and configurations.
- Install and configure the self-hosted runners on each EC2 instance or virtual machine by following GitHub's documentation on setting up self-hosted runners.
-
Configure runner groups
- Identify logical units or categories for your self-hosted runners based on your infrastructure and requirements. For example, you could have different groups for different machine types, such as small, medium, and large.
- Define runner groups within your GitHub repository settings. Specify the number of concurrent jobs that each group can handle. This will determine the scaling behavior of your runners.
-
Assign labels to runners
- Assign labels to your self-hosted runners based on their characteristics. For example, you could use labels like "small", "medium", "large", or other relevant tags.
- Label each runner based on its capabilities, available resources, or geographical location. Labels will be used to categorize and identify runners for specific workflows.
-
Create or modify workflows
- In your GitHub Actions workflows, specify the runner groups and labels that should be used for each job. This can be done in the YAML file defining your workflow.
- Update your workflows to utilize the self-hosted runner groups and labels, allowing GitHub Actions to select the appropriate runners for each job.
-
Adjust runner group concurrency
- Determine the desired concurrency settings for each runner group based on your workload and available resources.
- Configure the concurrency settings for each runner group in the repository settings on GitHub. Adjust the concurrency to increase or decrease the number of jobs that can run simultaneously in each group.
-
Monitor and adjust
- Monitor the performance and utilization of your self-hosted runners, both within the EKS cluster and on the EC2 instances or virtual machines.
- Utilize GitHub's monitoring and metrics to understand resource usage and identify potential bottlenecks or areas for improvement.
- Based on your observations, make adjustments to the scaling settings, such as increasing or decreasing the concurrency or adding/removing runners from a group.