Document deployment patterns & configuration

[Paul Beaudoin]

I'd like to continue the discussion started in NYPL/discovery-api#93 , which proposed placing common deployment config in a central place like engineering-general. There are several patterns for deployment emerging from our work, with lots of common config. I like @nodanaonlyzuul's idea that we collect that information in a central place, perhaps organized by technology (e.g. EB, lambda, EKS). For any kind of deployment, we could document the distinct strategies we consider best practice and link to a boilerplate repo, yeoman generator, etc. This would help bring up new apps, but could also allow us to slim down our READMEs by letting us link directly to the deployment standard the app uses, reducing the app's own responsibility for documenting all of the relevant eb subcommands, for example. (Separately, but in that same spirit, I'd love for us to develop a vocabulary of git deploy strategies to make it easier to read at a glance what kind of app we're dealing with.)

[Stephen Tiberius Schor]

I'm glad we're still talking about this.

Initial Configuration

We have some framework specific documentation already (e.g. elastic beanstalk / lambdas).
Do you have any thoughts on how that could be improved?

Deployment & Git

I'm trying to square our desire to document common patterns with our growing use of CI / CD pipelines. I think CD means WAY fewer README entries saying "type this to deploy" because humans won't be deploying. But we should have documentation somewhere saying "your travis.yml should do this..."

Regarding git I've been trying to document an app's git strategy along with its deployment for apps that use CD. (CD makes that important since merging into certain branches is tantamount to deploying to certain environments. Here's my first shot at it.

The more I've been using Travis for CD the more I've been thinking of human deployments like I do SSH access, something developers can do but shouldn't have to. I'd like us to strive for always doing CD. I know that's a big deal.

[Lucas Cioffi]

@nodanaonlyzuul @nonword if either of you would like to sit down and synchronously co-draft a rough Google Doc to think through some of this, I'd be happy to participate.

[Stephen Tiberius Schor]

Maybe we can chat about this over lunch this week?

[Ho-Ling Poon]

Been working on documenting Travis CI and what it does for deployment to Elastic Beanstalk. No PR yet as the instructions may get a little long.
https://github.com/NYPL/engineering-general/blob/travis-ci/standards/travis-ci.md

[Paul Beaudoin]

That's looking great, ho-ling. I've been trying to work out the best way to store travis-encrypted aws creds in .travis.yml. This is the stub documentation I've come up with. Is this more or less how we've been storing creds in other apps you've seen?

---

To configure Travis to deploy to EB:

• If you haven't already, log in to Github through travis:

   travis login --org
  

• Look up access keys and secrets for each AWS account you'll want to deploy to. Use low-privilege accounts where possible (e.g. nypl-sandbox "travis-deployer" account)
• Add encrypted KEY and SECRET variables to .travis.yml (note that if you're deploying to multiple AWS accounts, it may be useful to add account suffixes to differentiate pairs, like "_DEVELOPMENT" below):

   travis encrypt AWS_ACCESS_KEY_ID_DEVELOPMENT=[travis-deployer key id] --add
   travis encrypt AWS_SECRET_ACCESS_KEY_DEVELOPMENT=[travis-deployer secret] --add
  

• Add deploy entries to .travis.yml for each deploy hook, e.g.:

   deploy:
    - provider: elasticbeanstalk
      skip_cleanup: true
      access_key_id: "$AWS_ACCESS_KEY_ID_DEVELOPMENT"
      secret_access_key: "$AWS_SECRET_ACCESS_KEY_DEVELOPMENT"
      region: us-east-1
      app: discovery-api
      env: discovery-api-dev
      bucket_name: elasticbeanstalk-us-east-1-224280085904
      bucket_path: discovery-api-dev
      on:
        repo: NYPL-discovery/discovery-api
        branch: development
  

[Ho-Ling Poon]

@nonword, you complete me. Or to be more precise, this is what I'm looking for to include on the page. Feel free to commit that to the travis-ci branch where you see fit.

At our meeting, we should also talk about what should be include or anything that may be too long and cumbersome to include in the documentation.

[Kevin Friedman]

Also, there is a command documented in Parameter Store (/production/travis/add_aws) that automatically adds the necessary credentials to travis.yml. You can access it here:

https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Parameters:Name=%5BEquals%5Dproduction/travis/add_aws;sort=Name

[Paul Beaudoin]

Boom. I assumed something like that must be documented somewhere, but didn't know where to look. Thanks, Kevin. That simplifies the instructions 00ebf2b

[Matthew Koch]

This issue is quite old, over a year. Closing, with a tag of archived.