This repository accompanies our technical blog, "We’re on Cloud (Build) 9 with this Dagster secret".
The goal of this repository is to demonstrate how one can use Cloud Build to continously deploy Dagster projects in Google Cloud Platform.
To get started, make sure you have Docker installed.
We provided Makefile recipes to quickly run the Dagster project locally:
make compose-upTo shut down the project:
make compose-downThis repository is divided into three main parts:
- src/: contains all of our Dagster configuration files and application/pipeline code.
- deploy/: contains all of the Docker files that will be utilized by our Cloud Build instructions.
- ./: contains our Cloud Build instruction file and other Dockerfiles
Kindly note the following files:
- deploy/compose.Dockerfile is what Cloud Build will use to build a one-off container image that already has docker-compose installed.
- deploy/entrypoint.sh will be used to simply run the
docker-composecommands as soon as the container starts in our GCE instance
- deploy/entrypoint.sh will be used to simply run the
- daemon_dagit.Dockerfile will create an image that contains the code and configurations for long-running services such as Dagit, Dagster Daemon, and the gRPC server.
- This image will also store two key Dagster configuration files: workspace.yaml and dagster.yaml.
- pipelines.Dockerfile will create an image that contains all of our pipeline code. This image will be reused for every new pipeline run.
Expect to enable the following APIs/services in GCP:
- Container Registry
- Cloud Build
- Secret Manager
- Compute Engine
- OS Login (optional but recommended)
For the actual deployment, we leveraged a Google Compute Engine (GCE) instance with 2 vCPU and 2 GB memory (e2-small). We also leveraged the Container-Optimized OS provided by Google (cos-stable-97-16919-29-40).