Description

This repository is the definition of OpenMRS community infrastructure (infra-as-code). We are using terraform to generate network infra, Openstack VMs (Jetstream), DNS and backup resources.

You can also check https://github.com/bmamlin/jetstream-api-cli.git to check the results in Openstack.

This repository can generate documentation for all VMs created, and it's deployed to https://docs.openmrs.org/infrastructure/vms.html and https://docs.openmrs.org/infrastructure/vms.json

Check Provision new machine and Guidelines for New Servers for more details on how to create machines.

Requirements

Credentials

Before you can use this repository, you need:

Jetstream credentials - check internal wiki for Jetstream access details
Be included in git crypt in this repository (to access secrets)
Follow docs to obtain your personal openrc.sh file.

Software

You need to have installed:

git-crypt
ruby (2.0+)
thor (gem install thor)
wget

Development environment setup

To install terraform, run git crypto and initial setup (needed only once after cloning repo)

$ ./build.rb install

# edit conf/openrc-personal and add your Jetstream 2 credentials to 
$ vi conf/openrc-personal

# verify file is not encrypted
$ cat conf/openrc

# to init terraform in all VMs (takes time)
$ ./build.rb init_all

# to init terraform in a single VM (e.g. narok)
$ ./build.rb init narok

To undo the changes from the previous commands:

$ ./build.rb clean

To create a new stack test:

$ ./build.rb create test

To run terraform plan (and see what changed on your stack) on a base-network stack:

$ ./build.rb plan base-network/

To run terraform apply (and apply changes to a stack) on a base-network stack:

$ ./build.rb apply base-network/

To see all available commands:

$ ./build.rb

Forcing a VM to be reprovisioned (and keep the data volume):

$ ./build.rb taint-vm <stack>
$ ./build.rb plan <stack>
$ ./build.rb apply <stack>

To SSH a machine before running ansible:

$ ssh -i conf/provisioning/ssh/terraform-api.key root@<machine>

After ansible, you should use your regular user.

To completely destroy a VM (and its data volume):

Edit modules/single-machine/vm.tf and edit prevent_destroy = true to prevent_destroy = false
Run:

$ ./build.rb destroy <stack>

Undo changes to modules/single-machine/vm.tf

To generate and update VM documentation:

$ ./build docs
$ ./build plan docs
$ ./build apply docs

Run OpenStack commands

Use the embedded Jetstream2 command line interface tool to access Jetstream2 through the command line. OpenStack commands may be issued using the "os" alias (e.g., os server list).

$ cd cli
$ ./cli.sh

This will invoke a docker container with the necessary settings to interact directy with Jetsteam's OpenStack environment via the command line.

Repository organisation

build.rb: build helper (thor) file
conf/ : configuration files and authentication files
conf/template-stack: base files used when creating new stacks
conf/provisioning: keys and helpers to run ansible when provisioning a machine. Each
modules/: terraform modules
modules/single-machine: terraform module to generate a machine, with a A DNS record.
global-variables.tf: global terraform variables symlinked on each stack
base-network/ : stack basic infrastructure (network, subnets, routers)
other-stacks/: each machine should have a directory defined in here. Each folder should be a different state/stack file.

Each stack should be more or less:

main.tf: terraform resources
outputs.tf: stack outputs (can be used by other stacks as input params)
variables.tf: terraform variables
global-variables.tf: symlink to the top level global file.

Troubleshooting

Could not create DNS entries

Verify that the entry doesn't already exist in our DNS provider.

Cannot run null_resources via SSH after first ansible run

Ansible configures and secures our SSH configuration, so root cannot SSH anymore.

Change ./global-variables and use your username and (passphrase-less) key. Do not commit this change.

Gotchas

DNS CNAME records cannot be imported by terraform, so they have to be deleted in our DNS server before using them in a stack.
Don't use the DNS redirect. It doesn't support HTTPS.
Use /etc/ansible/facts.d/ files to export data to ansible. If the files should be modified, you can do it manually.

Resources needed by Terraform

Some resources are necessary to run terraform, so they were created manually:

S3 bucket to keep terraform state (versioned)
User to interact with bucket (access via bucket policy)
DNS domains (not defined in terraform provider)

openmrs / openmrs-contrib-itsm-terraform