SGTM

One-way sync of GitHub pull requests to Asana tasks so engineers can track all of their work in Asana. To see a more detailed explanation of the functionality of SGTM, see the code_reviews docs.

Setup

Follow these instructions for setting up SGTM to run in your environment and your infrastructure! Note that this is currently only set up for deployment on AWS, so if you are using a cloud provider, you may need to modify some code and deploy the app yourself.

Fork repository and set up your local repository

You will need to set some overrides specific to your deployment -- mostly due to the fact that AWS S3 bucket names are globally unique, but you may want to tweak some default configuration settings. So, we recommend forking this repository into your Github organization.

Installation

We recommend setting up a virtual environment to install and run your python environment. By doing so, you can eliminate the risk that SGTM's python dependencies and settings will be mixed up with any such dependencies and settings that you may be using in other projects. Once you have that activated (see Installing a Virtual Environment for Python below), you should install all required python dependencies using pip3 install -r requirements.txt -r requirements-dev.txt.

Install Terraform

You'll need to install Terraform to launch the infrastructure for SGTM.

Install Terragrunt

You'll need to install Terragrunt to configure Terraform for your own account.

Create your credentials for Asana/AWS/Github

There are three external services you'll need to interact with, and therefore need credentials for.

Asana

Create a Personal Access Token in Asana. At Asana, we created a Guest Account to run SGTM as, so no engineer's personal access token is used, and it's clear that there's a specific "SGTM" user who is making the task updates.

Copy this Personal Access Token for the next step.

AWS

You'll need to be able to authenticate with AWS via the command line, and there are a few ways to achieve that. See here for your options, but most likely you'll already have a preferred method of interacting with AWS via the command line.

Github

Again, you will probably want to create a new Github user in your org that is just for SGTM (since SGTM will be updating/merging PRs, it's clearer to attribute those actions to a user that is clearly name "SGTM" or something similar).

For the Github user you want to use, generate a Personal Access Token with the following permissions:
- repo (Full control of private repositories)
- read:org (Read org and team membership, read org projects)
Generate a secret token for your Github webhook. Github suggests generating this via ruby -rsecurerandom -e 'puts SecureRandom.hex(20)', but use whatever method you are comfortable with to generate a secure secret token. Save this somewhere, as you'll need it twice in the later steps.

Copy this Personal Accesss Token for the next step.

Create Asana Projects

You'll need to create two Asana projects: one that will store the mapping of Github username to Asana user id, and the other where your Github sync tasks will live.

Create your "SGTM Users" project (feel free to name this whatever you want -- this is just a suggestion). The requirements of this project are two custom fields named: "Github Username" (Text field) and "user_id" (Number field). Save the id of this project (from the URL once created) in ./terraform/terraform.tfvars.json under "asana_users_project_id".
To create your "SGTM tasks" project, use the setup_sgtm_tasks_project.py script. The script will prompt you for the PAT you generated earlier, and guide you through setting up a brand new project or updating an existing project with the recommended Custom Fields.
```
 >>> To setup a new project
 python3 scripts/setup_sgtm_tasks_project.py  -p "<PAT>" create -n "<PROJECT NAME>" -t "<TEAM ID>"

 >>> To update an existing project with the suggested custom fields
 python3 scripts/setup_sgtm_tasks_project.py  -p "<PAT>" update -e "<EXISTING PROJECT ID>"
```
1. If you have multiple repositories you want synced to Asana, you can create several of these projects. Make sure to take note of all of the project IDs for a later step.
2. If you are on Asana Basic and do not have access to Custom Fields, the script will skip that step - SGTM will work even without the suggested fields
Make sure that the Asana user/guest that you created earlier is a member of both of these projects.

Set your Terraform variables

NOTE: AWS S3 Bucket names are globally unique, so you will need to choose your own bucket names to be unique that no other AWS account has already created.

In ./terraform/variables.tf, any variable that is listed without a default value needs to be set. The preferred method of setting these values is through environment variables. For example, to se terraform variable asana_users_project_id, you'll want to set an environment variable TF_VAR_asana_users_project_id.
Save these somewhere that you and others collaborating on this deployment could share (we save ours in an Asana task internally, of course) since these will need to be the same each time you apply new changes.

Run setup script

You'll first need to set up the Terraform remote state to be the source of truth for the state of your deployed infrastructure.

Run python3 ./scripts/setup.py state (this will create an S3 bucket and DyanmoDb lock table for Terraform)
Initialize and apply the infrastructure:

> cd ./terraform
> terragrunt init
> terragrunt apply

Save the output of terragrunt apply, which should print out a api_gateway_deployment_invoke_url. You'll need this in the next step.
Push your secrets to the ecrypted S3 bucket that Terraform just created. cd back to the root of your repository and run: python3 ./scripts/setup.py secrets and follow the prompts.

Add Mapping of Github Repository -> Asana Project

For each repository that you are going to sync:

Find that repository's Github Graphql node_id:
1. You can get this using curl -i -u <username>:<github_personal_access_token> https://api.github.com/repos/<organization>/<repository>
Using the "SGTM tasks" project id from Create Asana Projects, update the sgtm-objects DynamoDb table with the mapping of {"github-node": "<node_id>", "asana-id": "<project_id>"}

Create Your Github Webhook

For each repository that you want to sync to Asana through SGTM:

Navigate to https://github.com/<organization>/<repository>/settings/hooks
Click "Add webhook"
Under "Payload URL", input the api_gateway_deployment_invoke_url from the previous step
Under "Content Type", select "application/json"
Under "Secret", input your secret token that you generated earlier
Under "Which events would you like to trigger this webhook?", select "Let me select individual events."
1. Issue comments
2. Pull requests
3. Pull request reviews
4. Pull request review comments
5. Statuses
Make sure "Active" is selected
Click "Add webhook"

Take it for a spin!

At this point, you should be all set to start getting Pull Requests synced to Asana Tasks. Open up a Pull Request, and Enjoy!

Optional Features

SGTM has a few optional power features that are disabled by default, but can be enabled with environment variables.

Auto-merge pull requests

SGTM can merge your pull requests automatically when certain conditions are fulfilled. This behavior is controlled by adding labels to the PR in Github. If this feature is enabled, there are 3 different labels you can apply to your PR to cause the PR to be auto-merged under different conditions:

🔍 merge after tests and approval: auto-merge this PR once tests pass and the PR is approved
🧪 merge after tests: auto-merge this PR once tests pass (regardless of approval status)
🚢 merge immediately: auto-merge this PR immediately

In all cases, a PR with merge conflicts will not be auto-merged.

How to enable:

Set an env variable of TF_VAR_sgtm_feature__automerge_enabled to true
Create labels in your repository of merge after tests and approval, merge after tests and merge immediately

Auto-complete linked tasks

At Asana, pull requests often have corresponding Asana tasks that can be completed when the pull request merges. With this feature enabled, setting a Github label of complete tasks on merge on a PR will automatically complete any linked Asana tasks. Asana tasks can be linked by adding their URLs to a line under the string Asana tasks: in the PR description, as demonstrated below:

Asana tasks:
<task_to_complete_url> <another_task_to_complete_url>

How to enable:

Set an env variable of TF_VAR_sgtm_feature__autocomplete_enabled to true
Create a label of complete tasks on merge in your repository

Note: If the SGTM user in your Asana domain doesn't have access to a linked task, it won't be able to merge it. You can add the SGTM user as a collaborator on a task to give it the ability to auto-complete the task.

Select users for follow-up review

SGTM can avoid closing tasks if the approvals come from certain Github users. This can be useful if you have specific Github users that you would like to be able to approve PRs in order to unblock merging, but that you want a second set of eyes on. For example, you may have a bot that automatically approves certain auto-generated PRs to speed up some workflow, but you still want a human to review those changes afterwards.

How to configure:

Set an env variable of TF_VAR_sgtm_feature__followup_review_github_users to contain a comma-separated list of Github usernames that should have follow-up review

Turn off Github team task subscriptions to reduce inbox noise

By default SGTM subscribes every member of a reviewing Github team to the SGTM task. You might want to turn this off if you want to use team reviewers as a marker for PR ownership or triage, but don't need every member of that team to see each PR.

How to configure:

Set an env variable of TF_VAR_sgtm_feature__disable_github_team_subscription to true

Always set the Asana task assignee to be the author of the Github Pull Request

By default SGTM will assign the task corresponding to the PR to the assignee on the Github pull request. The assignee on the Asana task will change for the following events: when the PR is in draft status, when changes are requested on the PR, when the PR is approved, or when a review is requested. Turning this feature on will keep the assignee of the task corresponding to the PR to always be the author of the pull request. This is useful if you want to reduce Asana inbox noise from reassignment or if you prefer to have the author of the PR be responsible for the corresponding Asana task.

How to configure:

Set an env variable of TF_VAR_sgtm_feature__allow_persistent_task_assignee to true
Ensure that the PR has the persistent task assignee Github label attached to the PR.

Rerun required checks on pull requests that are older than N hours with a specific base ref

SGTM can use Github API to rerun Check Runs on pull requests with a specified base ref if the results of those check runs exceeds a set number of hours. This is useful for keeping the status of your check runs "fresh" especially if the base ref is updated frequently. It will ignore pull requests that do not match the specified base ref.

Note: This does not use Github's check conclusion state stale.

How to configure:

Set an env variable of TF_VAR_sgtm_feature__check_rerun_base_ref_names to contain a comma-separated list of ref names (e.g. master, main) that pull requests must be based off of to have their check runs rerequested. The default is "main,master".
Set an env variable of TF_VAR_sgtm_feature__check_rerun_threshold_hours to any positive integer to represent the number of hours before a check run will be rerequested. The default is 0 which disables the feature.

Installing a Virtual Environment for Python

See these instructions for help in setting up a virtual environment for Python, or use the following TL;DR version:

run python3 -m venv v-env to create a virtual environment
run source v-env/bin/activate to activate and enter your virtual environment
once activated, run deactivate to deactivate and leave your virtual environment

Manual Testing

Because SGTM doesn't currently support a "staging" deployment to test changes, manual testing is still recommended for changes you will be making. Here are step-by-step instructions on how to test manually/locally:

Create a Personal Access Token in Asana. Copy that token and export it in your shell environment (export ASANA_API_KEY=<your_asana_personal_access_token>)
Create a Github Personal Access Token as per the instrucitons in the Github section above. Export that token in your shell environment (export GITHUB_API_KEY=<your_github_personal_access_token>)
Follow the instructions in Installing a Virtual Environment for Python, and then after activating the virtual environment, pip install -r requirements.txt -r requirements-dev.txt
Open up a python REPL in the SGTM root directory (or use ipython, but you'll have to pip install ipython first)
Run the function you want to test. It's usually fine / recommended to skip the DynamoDb locking when testing locally, since you usually won't be needing to test that. Here's an example of how to test updating a pull request:
1. Note what code you want to test. In this case, we want to go to src/github/webhook.py and look at _handle_pull_request_webhook. It looks like we need an pull_request_id.
2. Get the pull_request_id. One easy way to do this is to run a command like this curl -i -u <github_username>:$GITHUB_API_KEY https://api.github.com/repos/asana/sgtm/pulls/123 and then grab the node_id from that response.
3. Open up your REPL. Import the function you want to test (in this case: import src.github.controller as github_controller; import src.github.graphql.client as graphql_client)
4. Run the code! In this case:
  1. pull_request = graphql_client.get_pull_request(<pull_request_id>)
  2. github_controller.upsert_pull_request(pull_request)

Running Tests

To run the tests, you must set the AWS_DEFAULT_REGION environment variable. This is required because some of the tests are integration tests that require DynamoDb. This needs to be exported, so that it is available to sub-processes. Here's how:

if [ -z "$AWS_DEFAULT_REGION" ]; then export AWS_DEFAULT_REGION="us-east-1"; else export AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION; fi

You may then run all tests via the command line:

python3 -m unittest discover

Alternatively, you may run specific tests e.g. via:

python3 -m unittest test/<python-test-file-name>.py
python3 -m unittest test.<python-test-module-name>.<TestClassName>
python3 -m unittest test.<python-test-module-name>.<TestClassName>.<test_function_name>

"Building"

Please perform the following checks prior to pushing code

run black . to autoformat your code
run mypy on each file that you have changed
run tests, as described in the previous section

Asana / SGTM

SGTM

Setup

Fork repository and set up your local repository

Installation

Install Terraform

Install Terragrunt

Create your credentials for Asana/AWS/Github

Asana

AWS

Github

Create Asana Projects

Set your Terraform variables

Run setup script

Add Mapping of Github Repository -> Asana Project

Create Your Github Webhook

Take it for a spin!

Optional Features

Auto-merge pull requests

Auto-complete linked tasks

Select users for follow-up review

Turn off Github team task subscriptions to reduce inbox noise

Always set the Asana task assignee to be the author of the Github Pull Request

Rerun required checks on pull requests that are older than N hours with a specific base ref

Installing a Virtual Environment for Python

Manual Testing

Running Tests

"Building"

About

Languages