Investigate Tag-based Deployment

[Mephistic]

In order to de-risk deploys, we should consider switching to a tag-based deployment model.

Our current deployment process has some sharp edges around reverting bad deploys - the revert process is a bit convoluted, which reduces confidence in our ability to revert bad deploys (increasing the risk level of deployments and reducing their cadence).

To alleviate this pain, we should consider an easier to grok tag-based deployment model:

• When we're ready to deploy, create a tag for the current head of main (git tag -a TAG_NAME) or (git tag -a TAG_NAME COMMIT_HASH to tag a specific non-HEAD commit)
• Push this new tag with git push origin TAG_NAME
• To deploy, use the TAG_NAME as an input to a manually triggered deploy action in GitHub
• To rollback a bad deploy, simply run the deploy action with a previous TAG_NAME

Using specific, named versions of the codebase should make deploys easier to understand. e.g.

• We're on v23
• We make some changes, create a new tag v24, and deploy that tag
• We see a bug and need to revert, so we simply redeploy v23 while we investigate the bug.

Success Criteria:

• Ability to deploy a tag instead of a branch
• All branch protection we have for the current prod branch apply to the new deploy tags
• Ability to rollback a deploy (e.g. deploy an older tag)

Potential Tradeoffs:

• The Github UI is more limited working with tags than branches. (e.g. Tags seem tied to Releases, and we don't necessarily need or want Releases). If we don't want to create Releases with each tag, we may need to manage tag creation through the command line (or whatever Git UI clients).
• It might be simpler to both deploys and rollbacks with just tag pushes, but by design, we can't re-push an existing tag. The manual action fits our deploy cadence and stays in the GitHub UI, so I don't think it's a dealbreaker - but it's worth calling out.