thundergolfer / example-bazel-monorepo

🌿💚 Example Bazel-ified monorepo, supporting Golang, Java, Python, Scala, and Typescript

Home Page:https://buildkite.com/thundergolfer-inc/the-one-true-bazel-monorepo

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Properly document how Buildkite + Bazel was setup

thundergolfer opened this issue · comments

Quick sketch of what I remember doing:

  • Create a new pipeline with a single step that just runs bazel test //...
  • Go the Buildkite (BK) agents page and select the AWS Quickstart guide. Click the button that auto-creates a CloudFormation stack in your account.
  • After that CloudFormation (CF) stack has spun up, it's likely that you'll be able to see some agents appear in your agents list in the BK UI, but they won't be able to run the step yet.
  • Follow the instructions here on how to create an SSH keypair that BK and Github will share so that your private code can be pulled
  • Create a bootstrap script like the boostrap_agent.sh script found in this repo that is able to install Bazel and its requirements (e.g Java SDK) on the BK-default 'Amazon Linux2' AMIs.
  • Upload that script into a new S3 bucket next to the S3 buckets buildkite created (like the one for private SSH keys).
  • Go into IAM in the AWS console and copy the policy that grants the buildkite-agent role access to the SSH keys bucket and modify it so that it grants that role access to the bucket that contains your bootstrap script.
  • If this is working correctly, then when you re-run BK pipeline tasks you should see Bazel correctly starting.
  • Go into the CloudFormation stack and click the "Update" button to update some parameters:
  • Update MaxInstances to 1 (it's cheaper)
  • Update MinInstances to 0 (it's cheaper)
  • Set BuildkiteTerminateInstanceAfterJobDecreaseDesiredCapacity and
    BuildkiteTerminateInstanceAfterJobtotrue`. This will scale down the ASG instances when BK jobs are finished, which is cheaper
  • Reduce the EBS volume size to 100GB, which should be plenty when starting out
  • Up the instance from t2-nano to at least t2-small, because otherwise you'll get out of memory errors.