hdeiner / zipster-Vault-AWS-Lambda-build-environments-on-demand

Deploying A Java microservice using MySQL for backend and Vault for server configuration in AWS Lambda.

Deploying A Java microservice using MySQL for backend and Vault for server configuration in AWS Lambda.

Concept

Why do we use Docker Containers, anyway? We want to be able to create environments, anything from simple short lived and test isolated environments and right through to production, using the same easy to follow techniques that reduce risk that deployments will go bad. We reduce this risk by making deployments (even deployment environments) an everyday occurance, and testing frequently.

This project does all of that, and demonstrates a reference solution that includes secrets management using Vault in the process, so we can elimate the tyranny of decentralized environment configuration, and make system secrets something that systems ask for, rather than being pushed onto an environments surface.

• Docker containers. Lots of them.
• Docker Swarm orchestration and load balancing for demonstration and non-functional testing against.
• Prometheus, Grafana, cAdvisor, Node Exporter, Alert Manager, and Unsee (for environment monitoring and alerting)
• Portainer (for container management and trouble shooting)
• Java (for our code)
• Spark (for running a small REST server)
• MySQL (for persistence of a zipcode database)
• FlyWay (to version control the database)
• Vault is a vital component of this project. We will use it for holding together environments, the endpoints in those environments,and the secrets needed for those environments.

Here's a sample of what zipster does. It searches for post offices within a given radius of a given zipcode.

---

Explanation of the scripts to run.

The scripts which comprise this project are grouped as follows.

1. creating the Docker Images we need for our Vault, MySQL, and Spark servers
2. running the images quickly and locally, such as for when we are developing and testing our code (tests are not included in this project for brevity)
3. running the images in AWS for three variants. Both start with Vault being run and available for everyone.
   • bringing up an "AWS-QA" environment. The name is just to distinguish it from other environments and to hint at the capabilities.
   • bringing up and "AWS-QA-ELB" environment. Again, this name implies how one can put together a clustered environment. Of course, using terraform, adding or removing additional EC2 instances from this are quite easy.
   • bringing up and "AWS-QA-ELB" environment. Again, this name implies how one can put together a clustered environment. Of course, using terraform, adding or removing additional EC2 instances from this are quite easy.

Now, let's go into depth for each script.

---

createImages

...
\

run_Locally

...




\

runAWS 01_init

...
\

runAWS 02_vault

...







[we will bring down Vault at the very end, when everyone is done using it]

runAWS 03_awsqa - one ec2 instance for MYSQL and one ec2 instance for Spark Zipster

Bring Up

...



Bring Up - Effect in AWS



Bring Up - Effect in Vault







Test



Bring Down

Bring Down - Effect in AWS



Bring Down - Effect in Vault


\

runAWS 04_awsqa_elb - one ec2 instance for MYSQL, two ec2 instancex for Spark Zipster, one AWS ELB

Bring Up

...



Bring Up - Effect in AWS




Bring Up - Effect in Vault








Test



Bring Down

...

Bring Down - Effect in AWS



Bring Down - Effect in Vault



\

runAWS 05_awsqa_swarm - one ec2 instance for MYSQL, two ec2 instances for Swarm Workers, one ec2 instance for Swarm Manager (plus portainern and prometheus and grafana)

Bring Up

...



Bring Up - Effect in AWS



Bring Up - Effect in Vault












Test



Test - Look at Manager Node



Test - Look at Portainer









Test - Look at Grafana






Bring Down

...

Bring Down - Effect in AWS



Bring Down - Effect in Vault







\

runAWS 02_vault [bring down]