Node Todo

A simple node app to test GCP stack deployments.

Requirements

I. Google Workspaces

Follow the instructions here to set up Google Workspaces.

Save the following details:

Organization Domain: This is generally the web domain you configured your Google Workspaces account with. Example: mydomain.com
Customer ID: This is the random ID associated with your GCP account. Details on how to find it here: https://support.google.com/cloudidentity/answer/10070793

II. Google Cloud Platform

Folder and Project Structure

The trick here is to organize projects into folders that allow us to have a highly privileged Terraform agent across all projects managed by IaC, while keeping it from being able to self-escalate its own permissions beyond its sandbox and thus throughout the entire GCP account.

# Folder / Project Structure and Service Accounts

📁 Terraform-Managed-Resources (Folder)
├── 📁 Terraform-Managed-Projects (Folder)
│   ├── 📁 My-App (Folder)
│   │   └── 🚀 My-App (Project)
│   │       ├── 👤 Resource1 (Service Account)
│   │       └── 👤 Resource2 (Service Account)
│   │    
│   ├── 📁 My-App-Dev (Folder)
│   │   ├── 🚀 My-App-Staging (Project)
│   │   │   └── 👤 Resource1 (Service Account)
│   │   │ 
│   │   ├── 🚀 My-App-PR-456 (Project)
│   │   │   └── 👤 Resource1 (Service Account)
│   │   │ 
│   │   └── 🚀 My-App-PR-123 (Project)
│   │       └── 👤 Resource1 (Service Account)
│   │     
│   │     
│   ├── 📁 Some-Other-App (Folder)
│   │   └── 🚀 Some-Other-App (Project)
│   │       └── 👤 Resource1 (Service Account)
│   │     
│   └── 📁 Some-Other-App-Dev (Folder)
│       ├── 🚀 Some-Other-App-Staging (Project)
│       │   └── 👤 Resource1 (Service Account)
│       │
│       └── 🚀 Some-Other-App-PR-678 (Project)
│           └── 👤 Resource1 (Service Account)
│
│
└── 🚀 terraform-agents
    └── 👤 Terraform (Service Account)

📁 Terraform-Managed-Resources (Folder): All Terraform-managed resources including the Terraform Agent Service Account are contained within a single root folder. This not only gives us the ability to keep these resources organized, but it gives us a way to give the Terraform Agent the high level of permissions it needs in order to manage project resources while keeping it contained within the account.

📁 Terraform-Managed-Projects (Folder): Within the root IaC folder, all projects are contained within a sub-folder. This allows us to create the service accounts required to manage the resources for a given app or environment without allowing them to escalate the scope of their permissions to the level of the Terraform Agent and thus edit or affect resources for an app or environment that they should not have access to.

🚀 Terraform-Managed-Projects (Project): Each app or environment will need a project to associate the service accounts required to manage its resources. Depending on system architecture needs over time, this could represent any number of applications, each of which will likely require at the very least one development and one production environment. For some applications, we may even want to automate the launch of unique environments on pull request "open" events.

🚀 terraform-agents (Project): Every service account needs a project. This project only exists in order to create a service account for the terraform agent that is privileged enough to manage all projects managed by Terraform, while limiting it to only IaC projects.

Stack

Static Front End

An example of this type of architecture can be found here

Using the React app in the examples folder of this repository, you may create a build and then push the static assets in /dist to the storage bucket to deploy the frontend.

Service Accounts

IAM Policies

Network

A good network module can be found here. This may be overkill for what we are trying to do but it's something to fall back on.

We can also use the default network to get started.

Firewall Rules

Load Balancing

Backend API

We will be using Cloud Run to orchestrate and manage container clusters.

Cloud Run Official Docs

Cloud Run GCP Console

Cloud Run Terraform Simple Example

Cloud Run Terraform Example with VPC

Cloud Run Simple Example Blog Post

Questions:

What inputs does a Cloud Run module need in order to launch successfully?

In-memory Cache

Database

Deployments

Use GitHub Actions workflows to test, build and deploy code changes.

Notes

Authenticating Terraform Cloud with GCP

You will need a service account for the Terraform agent to interact with GCP. There are mulitple ways to perform this. Ideally, what you want to do is use Workload Identity Federation. We're going to do a classic service account with a credentials json for speed and simplicity. In the future we should replace this with WIF. It's really not that hard.

Workload Identity Federation

This is a dump of actions, not necessarily meant as instructions. It's more of a log of what I'm doing. It will need to be refined once everything is working and ideally repeated for verification.

Terraform Cloud:

Create a Terraform Cloud workspace
Create a GCP Service Account
Create a GCP_CREDENTIALS variable in TF Cloud
Copy the Service Account credentials json and set the value of the GCP_CREDENTIALS to the contents of the json.
Copy the generated main.tf file from Terraform Cloud, particularly the config with the organization and workspaces values

terraform {
  cloud {
    organization = "cyberworld-builders"

    workspaces {
      name = "todo-test"
    }
    required_providers {
        google = {
        source  = "hashicorp/google"
        version = ">= 4.45.0"
        }
    }
  }
}

Tested with terraform plan command locally

Google Cloud:

Created a Terraform-Managed-Resources Folder

NOTE: there was a blog post i had open earlier that documented this as gcloud commands. This is the ideal method because it's much easier to document and source control than console navigation instructions. It sucks that I lost it because it was dope and it looked comprehensive.

Create a terraform project for shared resources.
- initially created the project on the root level with no organization
- create a service account under the terraform project
- copy the json to the

NEXT STEPS:

Create a demo project for testing.
Add roles to the servcie account
Set up CI for Terraform changes

(Need to determine the registry solution. Are we using the GitHub Registry or the GCP registry. Seems like GCP is easier to authenticate. GitHub is better for hosting NPM packages.)

Create a GitHub Action for building the api image with Docker

Command Reference

Many times, especially in the initial setup process, resources need to be created in GCP first and then imported into IaC (Terraform). The following are some useful commands for gcloud and terraform command line interfaces for launching and/or importing resources.

GCP CLI (gcloud)

Making a small number of basic one-time changes is often quick and simple to perform directly within the GCP console UI. Once you find that your changes are growing in number and complexity and/or need to be performed multiple times or reproduced; it makes more sense to use the CLI.

The CLI has the following advantages in this case:

It is easier to document, track and share precise instructions.
The console requires your browser to use a lot of your system memory and network bandwidth. Users often find that opening up mulitple tabs causes your machine to lag and screen shares can crash.
Navigating the console is often a more complex experience and can lead to more user errors.

Account Configuration Commands:

Additional information on setting gcloud account configs can be found here.

# List Configurations
gcloud config list

# Create Configuration
gcloud config configurations create config-name

# Set Project
gcloud config set project my-project-id

# Set Account
gcloud config set account my-account@example.com

# Activate Config (when managing mulitple accounts)
gcloud config configurations activate config-name

Create Folder and Project Structure:

# Create Root Folder
gcloud resource-manager folders create --display-name=Terraform-Managed-Resources --organization=ORGANIZATION_ID

# Create Terraform Agents Project
gcloud projects create terraform-agents --folder=IAC_ROOT_FOLDER_ID

# Create Terraform Managed Projects Folder
gcloud resource-manager folders create --display-name=Terraform-Managed-Projects --folder=IAC_ROOT_FOLDER_ID

TF CLI (terraform)

Once certain resources are created in the console they will need to imported back into the Terraform state so that they can be tracked by IaC.

jaylong255 / node-todo