Cloud Foundry Diego [BOSH release]

This repo is a BOSH release for deploying Diego and associated tasks for testing a Diego deployment. Diego builds out the new runtime architecture for Cloud Foundry, replacing the DEAs and Health Manager.

This release relies on a separate deployment to provide Consul, NATS and Loggregator. In practice these come from cf-release.

Additional Diego Resources

The Contribution Guidelines describes the developer workflow for making changes to Diego.
The Diego Design Notes present an overview of Diego, and links to the various Diego components.
The Migration Guide describes how developers and operators can manage a transition from the DEAs to Diego.
The Docker Support Notes describe how Diego runs Docker-image-based apps in Cloud Foundry.
The Diego-CF Compatibility Log records which versions of cf-release and diego-release are compatible, according to the Diego team's automated testing pipeline.
Diego's Pivotal Tracker project shows what we're working on these days.

BOSH Dependencies
Discovering a Set of Releases to Deploy
Deploying Diego to BOSH-Lite
Pushing to Diego
Deploying Diego to AWS
Database Encryption
Configuring Encryption Keys
TLS Configuration
Generating TLS Certificates
Custom TLS Certificate Generation
Recommended Instance Types

BOSH Dependencies

When deploying diego-release via BOSH, the following minimum versions are required:

BOSH Release v206+ (Director version 1.3072.0)
BOSH Stemcell 3125+

These versions ensure that the pre-start script in the rootfses job will be run to extract and configure the cflinuxfs2 rootfs and that the drain scripts will be called for all jobs on each VM during updates, instead of only the first job.

Release Compatibility

Diego releases are tested against Cloud Foundry, Garden, and ETCD. Compatible versions of Garden and ETCD are listed with Diego on the Github releases page.

Checking out a release of Diego

The Diego git repository is tagged with every release. To move the git repository to match a release, do the following:

cd diego-release/
# checking out release v0.1437.0
git checkout v0.1437.0
./scripts/update
git clean -ffd

From a final release of CF

On the CF Release GitHub Releases page, recommended versions of Diego, Garden, and ETCD are listed with each CF Release. This is the easiest way to correlate releases.

Alternatively, you can use records of CF and Diego compatibility captured from automated testing. First look up the release candidate SHA for your CF release. This is listed as the commit_hash in the release yaml file. Find the SHA in diego-cf-compatibility/compatibility-v2.csv to look up tested versions of Diego Release, Garden, and ETCD.

Example: Let's say you want to deploy Diego alongside CF final release 222. The release file releases/cf-222.yml in the cf-release repository contains the line commit_hash: 53014242. Finding 53014242 in diego-cf-compatibility/compatibility-v2.csv reveals Diego 0.1437.0, Garden 0.308.0, and ETCD 16 have been verified to be compatible.

From a specific CF Release commit SHA

Not every cf-release commit will appear in the diego-cf compatibility table, but many will work with some version of Diego.

If you can't find a specific cf-release SHA in the table, deploy the diego-release that matches the most recent cf-release relative to that commit. To do this, go back through cf-release's git log from your commit until you find a Final Release commit and then look up that commit's SHA in the diego-cf compatibility table.

Deploying Diego to BOSH-Lite

Install and start BOSH-Lite, following its README. For garden-linux to function properly in the Diego deployment, we recommend using version 9000.69.0 or later of the BOSH-Lite Vagrant box image.
Upload the latest version of the Warden BOSH-Lite stemcell directly to BOSH-Lite:

bosh upload stemcell https://bosh.io/d/stemcells/bosh-warden-boshlite-ubuntu-trusty-go_agent

Alternately, download the stemcell locally first and then upload it to BOSH-Lite:

curl -L -o bosh-lite-stemcell-latest.tgz https://bosh.io/d/stemcells/bosh-warden-boshlite-ubuntu-trusty-go_agent
bosh upload stemcell bosh-lite-stemcell-latest.tgz

Please note that the consul_agent job does not set up DNS correctly on version 3126 of the Warden BOSH-Lite stemcell, so we do not recommend the use of that stemcell version.

Check out cf-release (runtime-passed branch or tagged release) from git:

cd ~/workspace
git clone https://github.com/cloudfoundry/cf-release.git
cd ~/workspace/cf-release
git checkout runtime-passed # do not push to runtime-passed
./scripts/update

Check out diego-release (master branch or tagged release) from git:

cd ~/workspace
git clone https://github.com/cloudfoundry-incubator/diego-release.git
cd ~/workspace/diego-release
git checkout master # do not push to master
./scripts/update

Install spiff according to its README. spiff is a tool for generating BOSH manifests that is required in some of the scripts used below.
Generate the CF manifest:

cd ~/workspace/cf-release
./scripts/generate-bosh-lite-dev-manifest

Or if you are running Windows cells along side this deployment, instead generate the CF manifest as follows:

cd ~/workspace/cf-release
./scripts/generate-bosh-lite-dev-manifest \
  ~/workspace/diego-release/manifest-generation/stubs-for-cf-release/enable_diego_windows_in_cc.yml

Generate the Diego manifests:

cd ~/workspace/diego-release
./scripts/generate-bosh-lite-manifests

EXPERIMENTAL: If using MySQL run the following to enable it on Diego:

cd ~/workspace/diego-release
USE_SQL='true' ./scripts/generate-bosh-lite-manifests

Create, upload, and deploy the CF release:

cd ~/workspace/cf-release
bosh deployment bosh-lite/deployments/cf.yml
bosh -n create release --force &&
bosh -n upload release &&
bosh -n deploy

EXPERIMENTAL: If configuring Diego to use MySQL, upload and deploy the latest cf-mysql-release:

cd ~/workspace/diego-release
bosh upload release https://bosh.io/d/github.com/cloudfoundry/cf-mysql-release
./scripts/experimental/generate-mysql-bosh-lite-manifest
bosh deployment bosh-lite/deployments/cf-mysql.yml
bosh -n deploy

Accessing the MySQL remotely:

You can access the mysql database used as deployed above with the following command:

```bash
mysql -h 10.244.7.2 -udiego -pdiego diego
```

Then commands such as `SELECT * FROM desired_lrps` can be run to show all the desired lrps in the system.

Upload the latest garden-linux-release OR garden-runc-release:

bosh upload release https://bosh.io/d/github.com/cloudfoundry-incubator/garden-linux-release

# if you specified [-g] when you generated your manifest:
# bosh upload release https://bosh.io/d/github.com/cloudfoundry-incubator/garden-runc-release

If you wish to upload a specific version of garden-linux-release, or to download the release locally before uploading it, please consult directions at bosh.io.

Upload the latest etcd-release:

bosh upload release https://bosh.io/d/github.com/cloudfoundry-incubator/etcd-release

If you wish to upload a specific version of etcd-release, or to download the release locally before uploading it, please consult directions at bosh.io.

Upload the latest cflinuxfs2-rootfs-release:

bosh upload release https://bosh.io/d/github.com/cloudfoundry/cflinuxfs2-rootfs-release

If you wish to upload a specific version of cflinuxfs2-rootfs-release, or to download the release locally before uploading it, please consult directions at bosh.io.

Create, upload, and deploy the Diego release:

cd ~/workspace/diego-release
bosh deployment bosh-lite/deployments/diego.yml
bosh -n create release --force &&
bosh -n upload release &&
bosh -n deploy

cf login -a api.bosh-lite.com -u admin -p admin --skip-ssl-validation &&
cf enable-feature-flag diego_docker

Now you are configured to push an app to the BOSH-Lite deployment, or to run the Smoke Tests or the CF Acceptance Tests.

If you wish to run all of the diego jobs on a single VM, you can replace the manifest-generation/bosh-lite-stubs/instance-count-overrides.yml stub with the manifest-generation/bosh-lite-stubs/colocated-instance-count-overrides.yml stub.

Pushing a CF Application to the Diego backend

Create and target a CF org and space:

cf api --skip-ssl-validation api.bosh-lite.com
cf auth admin admin
cf create-org diego
cf target -o diego
cf create-space diego
cf target -s diego

Change into your application directory and push your application without starting it:

cd <app-directory>
cf push my-app --no-start

Enable Diego for your application.
Start your application:

cf start my-app

Deploying Diego to AWS

In order to deploy Diego to AWS follow these instructions. Enjoy!

Database Encryption

Diego Release must be configured with a set of encryption keys to be used when encrypting data at rest in the ETCD database. To configure encryption the diego.bbs.encryption_keys and diego.bbs.active_key_label properties should be set.

Diego will automatically (re-)encrypt all of the data stored in ETCD using the active key upon boot. This ensures an operator can rotate a key out without having to manually rewrite all of the records.

Configuring Encryption Keys

Diego uses multiple keys for decryption while allowing only one for encryption. This allows an operator to rotate encryption keys in a downtime-less way.

For example:

properties:
  diego:
    bbs:
      active_key_label: key-2015-09
      encryption_keys:
        - label: 'key-2015-09'
          passphrase: 'my september passphrase'
        - label: 'key-2015-08'
          passphrase: 'my august passphrase'

In the above, the operator is configuring two encryption, and selecting one to be the active. The active key is the one used for encryption while all the other can be used for decryption.

The key labels must be no longer than 127 characters, while the passphrases have no enforced limit. In addtion to that, the key label must not contain a : (colon) character, due the way we build command line flags using : as a separator.

##TLS Configuration

Diego Release can be configured to require TLS for communication with etcd. To enable or disable TLS communication with etcd, the etcd.require_ssl and diego.bbs.etcd.require_ssl properties should be set to true or false. By default, Diego has require_ssl set to true. When require_ssl is true, the operator must generate TLS certificates and keys for the etcd server and its clients.

TLS and mutual authentication can also be enabled between etcd peers. To enable or disable this, the etcd.peer_require_ssl property should be set to true or false. By default, Diego has peer_require_ssl set to true. When peer_require_ssl is set to true, the operator must provide TLS certificates and keys for the cluster members. The CA, server certificate, and server key across may be shared between the client and peer configurations if desired.

Similarly, TLS with mutual authentication can be enabled for communication to the BBS server, via the diego.bbs.require_ssl and diego.CLIENT.bbs.require_ssl BOSH properties. These properties default to true. When enabled, the operator must provide TLS certificates and keys for the BBS server and its clients (other components in the Diego deployment).

Generating TLS Certificates

For generating TLS certificates, we recommend certstrap. An operator can follow the following steps to successfully generate the required certificates.

Most of these commands can be found in scripts/generate-diego-ca-certs, scripts/generate-etcd-certs, and scripts/generate-bbs-certs

Get certstrap

go get github.com/square/certstrap
cd $GOPATH/src/github.com/square/certstrap
./build
cd bin

Initialize a new certificate authority.

$ ./certstrap init --common-name "diegoCA"
Enter passphrase (empty for no passphrase): <hit enter for no password>

Enter same passphrase again: <hit enter for no password>

Created out/diegoCA.key
Created out/diegoCA.crt

The manifest properties properties.diego.etcd.ca_cert and properties.diego.bbs.ca_cert should be set to the certificate in out/diegoCA.crt.

Create and sign a certificate for the etcd server.

$ ./certstrap request-cert --common-name "etcd.service.cf.internal" --domain "*.etcd.service.cf.internal,etcd.service.cf.internal"
Enter passphrase (empty for no passphrase): <hit enter for no password>

Enter same passphrase again: <hit enter for no password>

Created out/etcd.service.cf.internal.key
Created out/etcd.service.cf.internal.csr

$ ./certstrap sign etcd.service.cf.internal --CA diegoCA
Created out/etcd.service.cf.internal.crt from out/etcd.service.cf.internal.csr signed by out/diegoCA.key

The manifest property properties.etcd.server_cert should be set to the certificate in out/etcd.service.cf.internal.crt. The manifest property properties.etcd.server_key should be set to the certificate in out/etcd.service.cf.internal.key.

Create and sign a certificate for etcd clients.

$ ./certstrap request-cert --common-name "clientName"
Enter passphrase (empty for no passphrase): <hit enter for no password>

Enter same passphrase again: <hit enter for no password>

Created out/clientName.key
Created out/clientName.csr

$ ./certstrap sign clientName --CA diegoCA
Created out/clientName.crt from out/clientName.csr signed by out/diegoCA.key

The manifest property properties.etcd.client_cert should be set to the certificate in out/clientName.crt. The manifest property properties.etcd.client_key should be set to the certificate in out/clientName.key.

Create and sign a certificate for the BBS server.

$ ./certstrap request-cert --common-name "bbs.service.cf.internal" --domain "*.bbs.service.cf.internal,bbs.service.cf.internal"
Enter passphrase (empty for no passphrase): <hit enter for no password>

Enter same passphrase again: <hit enter for no password>

Created out/bbs.service.cf.internal.key
Created out/bbs.service.cf.internal.csr

$ ./certstrap sign bbs.service.cf.internal --CA diegoCA
Created out/bbs.service.cf.internal.crt from out/bbs.service.cf.internal.csr signed by out/diegoCA.key

The manifest property properties.diego.bbs.server_cert should be set to the certificate in out/bbs.service.cf.internal.crt. The manifest property properties.diego.bbs.server_key should be set to the certificate in out/bbs.service.cf.internal.key.

Create and sign a certificate for BBS clients.

$ ./certstrap request-cert --common-name "clientName"
Enter passphrase (empty for no passphrase): <hit enter for no password>

Enter same passphrase again: <hit enter for no password>

Created out/clientName.key
Created out/clientName.csr

$ ./certstrap sign clientName --CA diegoCA
Created out/clientName.crt from out/clientName.csr signed by out/diegoCA.key

For each component CLIENT that has a BBS client, the manifest properties properties.diego.CLIENT.bbs.client_cert should be set to the certificate in out/clientName.crt, and the manifest properties properties.diego.CLIENT.bbs.client_key should be set to the certificate in out/clientName.key.

(Optional) Initialize a new peer certificate authority.

$ ./certstrap --depot-path peer init --common-name "peerCA"
Enter passphrase (empty for no passphrase): <hit enter for no password>

Enter same passphrase again: <hit enter for no password>

Created peer/peerCA.key
Created peer/peerCA.crt

The manifest property properties.etcd.peer_ca_cert should be set to the certificate in peer/peerCA.crt.

(Optional) Create and sign a certificate for the etcd peers.

$ ./certstrap --depot-path peer request-cert --common-name "etcd.service.cf.internal" --domain "*.etcd.service.cf.internal,etcd.service.cf.internal"
Enter passphrase (empty for no passphrase): <hit enter for no password>

Enter same passphrase again: <hit enter for no password>

Created peer/etcd.service.cf.internal.key
Created peer/etcd.service.cf.internal.csr

$ ./certstrap --depot-path peer sign etcd.service.cf.internal --CA diegoCA
Created peer/etcd.service.cf.internal.crt from peer/etcd.service.cf.internal.csr signed by peer/peerCA.key

The manifest property properties.etcd.peer_cert should be set to the certificate in peer/etcd.service.cf.internal.crt. The manifest property properties.etcd.peer_key should be set to the certificate in peer/etcd.service.cf.internal.key.

Custom TLS Certificate Generation

If you already have a CA, or wish to use your own names for clients and servers, please note that the common-names "diegoCA" and "clientName" are placeholders and can be renamed provided that all clients client certificate. The server certificate must have the common name etcd.service.cf.internal and must specify etcd.service.cf.internal and *.etcd.service.cf.internal as Subject Alternative Names (SANs).

Recommended Instance Types

If you are deploying to AWS, you can use our recommended instance types by spiff merging your iaas-settings.yml with our provided manifest-generation/misc-templates/aws-iaas-settings.yml:

spiff merge \
	manifest-generation/misc-templates/aws-iaas-settings.yml \
	/path/to/iaas-settings.yml \
	> /tmp/iaas-settings.yml

You can then use the template generated as the iaas-settings.yml for the scripts/generate-deployment-manifest tool. The cell jobs currently use r3.xlarge as their instance_type. For production deployments, we recommend that you increase the ephemeral_disk size. This can be done by specifying the following in your iaas-settings.yml under the cell resource pool definitions:

ephemeral_disk:
  size: 174_080
  type: gp2

nelsam / diego-release