C00ldudeNoonan / simple-dbt-runner

Simple dbt Runners

dbt Cloud: who needs it? With this repo, not you!

This repo gives you the ability to run dbt in production using GitHub Actions. There are several basic GH Action workflows you can take and modify for your needs:

• run dbt commands on a schedule
• run dbt after merging in a PR to the main branch (we recommend only choosing one of these at a time)
  • full run
  • state-aware run (only modified models)
• dbt CI runs on PR commits to make sure your changes will work

The state-aware workflow will look for the manifest.json file in a branch called gh-pages. We also take advantage of that branch to host your project's documentation website.

How To Set Up Your dbt Project

1. Fork this repo and copy your whole dbt project into the project_goes_here folder.
2. Create a Personal Access Token with Workflows (Read/Write) permission and add it to the repository action secrets with key WORKFLOW_TOKEN
3. Update your repository settings to allow GitHub Actions to create PRs. This setting can be found in a repository's settings under Actions > General > Workflow permissions. This is what it should look like.
4. Go to the Actions tab and run the Project Setup workflow, making sure to select the type of database you want to set up
   • This opens a PR with our suggested changes to your profiles.yml and requirements.txt files.
   • We assume if you're migrating to self-hosting you need to add a prod target to your profiles.yml file, so this action will do that for you and also add the db driver you indicate.
   • FYI we also assume you have a profiles.yml file.
5. Add some environment variables to your GitHub Actions secrets in the Settings tab. You can see which vars are needed based on anything appended with ${{ secrets. in the open PR. You might need to slightly edit this PR based on your project setup.
6. Run the Manual dbt Run to test that you're good to go.
7. Edit the Actions you want to keep and delete the ones you don't

dbt Documentation

dbt documentation is pushed to Github Pages. If you are using Github Enterprise, the pages are automatically secured behind Github SSO. Hosting your dbt docs is highly contextual based on your organization. There are proven patterns for shipping dbt docs to netlify, confluence and many other targets.

If using GH Pages, the only manual configuration required for hosting your dbt docs is to set it to run off the root directory of the gh-pages branch. You can configure this in your Github repo's Settings > Pages. Once you set that up it will looks like this

WARNING: if you do not have Gitub Enterprise and you set up the documentation hosting your page might be publicly accessible. Please review their docs.

Acknowledgements & Notes

Thank you to dwreeves for both highlighting an issue with initial deployment with regard to public s3 buckets as well as providing the template for deploying to github pages.