davidenq / go-cicd

Go CI/CD

Checklist

• unit testing with TDD
• integration testing
• end to end testing
• pipeline ci/cd using Travis-ci service (automatic deployment in different stages specified at .travis.yml file)
• deploy on local and cloud (google cloud run) (support docker-compose, google cloud run and google kubernetes engine)
• branching strategy based on Github Flow (Master for production, other branchs for development/staging)
• semantic versioning with git tags
• Infraestructure as a code with terraform and bash scripting for createing cluster on Kubernetes
• deploy docker containers on Google Kubernetes Engine through yaml files, kubectl and bash scripting
• Handle scripts as a centralized way through Makefile
• created a load balancer on Google Kubernetes Engine with 2 microservices
• Published Docker containers on Google Registry
• Code coverage using Coveralls service

This projects aims to show not only a process for CI/CD but also, to write code following clean code, TDD, branching strategy, versioning and so on. On the other hand, the folder structure follows the definition specified here. But it's not a standardized folder structure. However, big projects such as Kubernetes, Prometheus, and others, follow that folder structure.

List of Commands

The current implementation has a Makefile as a centralized handler of bash scripting. So for that reason, it's important to know what commands you can run

app command

The app command are used to work without Docker

Testing

Running unit and integration tests

$make app action=test

Running unit test

$make app action=unit-test

Running unit integration-test

$make app action=integration-test

To perform end-to-end testing, you must first run the application

$make app action=run

$make app action=e2e-test

Running the app

$make app action=run

docker command

docker command allows you to work based-on Docker.

Running the app

In order to work with Docker, you must first build de Docker image.

$make docker action=build

$make docker action=app

docker-compose command

docker-compose command just has two actions: up and down. The first one will build the Docker image and after that, will run the application. The second one, will turn off the Docker container.

$make docker-compose action=up

$make docker-compose action=down

travis command

Travis commands through make, are used for logging and for creating encrypt variables. Although those commands are not difficult to use, at first instance, those were added having in mind a ease way to use travis commands, but is a better option running Travis directly with its CLI. make travis commands will be deprecated for the next release.

gcr command

gcr command is used to deploy the Docker container on Google Cloud Run. It command is called by the Travis-CI service. You can not do anything in local with this command because there are environmental variables gotten from travis environment variables.

infra command

infra command is used to create a cluster on Google Kubernetes Engine. Terraform is used for providing this funcionality. This command is used only in local. For that reason, it's important that you can install gcloud CLI in your local machine, set the project of Google and provide credentials for authentication and authorization. This command doesn't provide you that funcionality (authentication and authorization).

For create a cluster:

$make infra

It's important to set all environment variables on ./iaac/terraform/gke/cluster.tfvars. You can get and example at ./iaac/terraform/gke/cluster.tfvars.example. For security, cluster.tfvars is ignoreg for git.

gke command

gke command is used to create the workloads on the cluster previously created. Bash scripting, kubectl and yaml files are used to automate this tasks.

$make gke service=[FOLDER_SERVICE_NAME]
#FOLDER_SERVICE_NAME is the name of folder localed at ./deploy/gke, you have two options servicea or serviceb
#example
$ make gke service=servicea
$ make gke service=serviceb

To expose those service, you need to create a ingress. You can execut the below command for that:

kubectly apply -f ./deploy/gke/ingress/service.yaml

Guidelines

1. Write unit testing (with TDD), integration testing and end to end testing a. For each unit test wrote, it must be committed to Github before writing the implementation. b. After that, must write the implementation and it will be tested in order to check whether it passed or not. When it has passed, the implementation must be committed to Github. c. Each implementation must be checked in order to improve the code following clean code and best development practices. After that, the new refactor must be committed to Github.
2. GitHubFlow as the strategy Branching

• Github Flow could be used in the first instance and only for test purposes in order to understand in a better way this strategy branching.

3. Semver for versioning

• It is important to generate a release only when you consider that the implementation could be completed to be tested.

Folder structure

The folder structure is based-on golang-layoud. This is not an official folder structure, but it is based-on the biggest projects such as Kubernetes.

• build folder contains files for building the application: For instance, as a binary, inside in a Docker Container an so on. Currently, Dockerfile is supported to build the server application on Docker.
• cmd contains the service applications called gocicd for this project. gocicd internally has a similarly folder structure of the clean architecture folder modularization but it not be implemented any pattern.
• deploy contains multiple definitions for deploy gocicd application not only in local with docker-compose, but also on cloud using Google Cloud Run and Google Kubernetes Engine.
  • docker-compose contains the definition in the YAML file
  • gcr contains a bash script to execute gcloud command to deploy on Google Cloud Run
  • gke contains multiple YAML files for create the workloads on Google Kubernetes Engine. There are 3 folders that contain:
    • configmap.yaml for creating the basic configuration with environment variables needed for the deployment.
    • deployment.yaml is used to create the deployment (multiples containers) for the gocicd application.
    • service.yaml is used to expose the deployment internally on Kubernetes ingress folder contains a service.yaml definition for creating a unique load balancer as a entrypoint for servicea and serviceb deployments

  note: each service is created through bash scripting using: make gke service=[SERVICE_NAME]

• iaac contains tools to create infraestructure as a code. Terraform is used to create the cluster on Google Kubernetes Engine.
• scriptsfolder is a centralized folder for creating all scripts for creating deployments, builds, iaacs, and so on.
• tests folder contains end to end testing. It is used after create a deployment on Google Cloud Run in order to test the whole application.
• account.json.enc is an encrypted file that contains the json auth account to login into Google. Just travis can be decrypt this file in the process of CI/CD.

To start running testing

For running unit testing:

$ make unit-test

To start to develop

Requirements

• Docker
• docker-compose
• ruby for travis cli
• go > 1.12 if you want to develop without Docker
• brew install goreleaser
• terraform

install travis

gem install travis

issues installing travis

• if you get 'You don't have write permissions for the /Library/Ruby/Gems/2.6.0 directory' running the below commands:

  $ echo 'export PATH="/usr/local/opt/ruby/bin:$PATH"' >> ~/.bash_profile

  $ source ~/.bash_profile

source

• if you get command not found, try:

  $ brew install travis

  travis encrypt-file credenciales.json --add