[info] CI on Gitlab with deployment to AWS EC2 instance

[Karel Mácha]

I host my project on Amazon AWS and the code on Gitlab. I managed to use yamup with Gitlab CI pipeline to do automatic deployment on Amazon AWS and I thought this might be interesting.

prerequisites

Normally, you need to configure a Security group on AWS specifying the firewall rules for accessing your (EC2) instances. The problem I had to target was how to (temporarily) allow SSH access for particular Gitlab CI runner. I use AWS command line interface for this.

In your Amazon IAM console you have to create user (which gives you the Access key ID and Secret access key) and this user need to have a policy attached, which allows security group management. I created a specific security policy for this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeTags",
                "ec2:DescribeSecurityGroups"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "ec2:RevokeSecurityGroupIngress",
                "ec2:AuthorizeSecurityGroupEgress",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:RevokeSecurityGroupEgress"
            ],
            "Resource": "arn:aws:ec2:*:*:security-group/*"
        }
    ]
}

Then create (locally) the necessary credentials and config files with the aws configure call as described here. The files will be created in the .aws directory in your home directory.

Next, you should store some variables in your Gitlab project (Settings - CI / CD - Variables) allowing storing your credentials and secrets outside of the CI script file. Mine looks like this:

• (file) AWS_SECURITY_GROUP_ADMIN_CONFIG - the content of the config file created in previous steps
• (file) AWS_SECURITY_GROUP_ADMIN_CREDENTIALS - the content of the credentials file created in previous steps
• (variable) AWS_SECURITY_GROUP_ID - the id of the AWS security group used by your AWS instance
• (file) AWS_VM_PEM - the content of the .pem file used by yamup
• (file) SETTINGS_JSON - the content of the meteor settings file
• (file) SM_YAMUP_JSON - the content of the yamup.json file

Gitlab CI script

Create an .gitlab-ci.yml file in your projects main directory describing the jobs and stages Gitlab would perform automatically. You can reference your variables, stored in Gitlab settings with $VARIABLE_NAME in your script and you can create symbolic links to the files stored in Gitlab settings with ln -s $FILE_NAME /path/to/location. My Gitlab .gitlab-ci.yml with the deployment script looks like this:

# use image with same distribution as the one running on your EC2 instance
image: ubuntu:18.04

before_script:
  # Install curl first
  - apt-get -qq update
  - apt-get install -qqy curl
  # Meteor Installation
  - curl https://install.meteor.com/ | sh
  - meteor npm install

test:
  script:
    # npm script in package.json `meteor test --once --driver-package meteortesting:mocha --allow-superuser`
    - meteor npm run test:as_root

deploy:
  only:
    # should be only triggered on master & release branches
    refs:
      - master
      - release

  script:
    # create symbolic links for aws cli credentials and config files
    - mkdir ~/.aws
    - ln -s $AWS_SECURITY_GROUP_ADMIN_CREDENTIALS ~/.aws/credentials
    - ln -s $AWS_SECURITY_GROUP_ADMIN_CONFIG ~/.aws/config
    # install standalone npm and aws cli from the repositories
    - DEBIAN_FRONTEND=noninteractive apt-get install -qqy npm awscli
    # save the current runner IP in a variable
    - RUNNER_IP=$(curl http://checkip.amazonaws.com)
    # allow SSH access from the current runner IP
    - aws --profile security-group-admin ec2 authorize-security-group-ingress --group-id $AWS_SECURITY_GROUP_ID --protocol tcp --port 22 --cidr $RUNNER_IP/32
    # initialize yamup
    - npm install -g yamup
    # meteor does not bundle directories starting with . 
    - mkdir .deploy
    - cd .deploy
    # create symbolic links for yamup config files
    - ln -s $SETTING_JSON ./settings.json
    - ln -s $YAMUP_JSON ./yamup.json
    # PEM file referenced in the yamup.json
    - ln -s $AWS_VM_PEM ./SSH-cert.pem
    - METEOR_ALLOW_SUPERUSER=true yamup deploy
    # cleanup
    - cd ..
    - rm -rf .deploy
    # revoke SSH access from the current runner IP
    - aws --profile security-group-admin ec2 revoke-security-group-ingress --group-id $AWS_SECURITY_GROUP_ID --protocol tcp --port 22 --cidr $RUNNER_IP/32
    - rm -rf ~/.aws

I hope this would be helpful to someone.