DevOps or Cloud-Native Challenge on Hybrid-Cloud

This is a challenge where you can showcase your technical skills and the considerations which you make when performing devops and infrastructure tasks. Your solution may be as simple or complex as you like, provided it meets the requirements below.

The PART 01

You are to create a web site which can be used for publishing blogs. As a proof of concept, it will be sufficient that you demonstrate a website that displays text (e.g. Lorem Ipsum https://en.wikipedia.org/wiki/Lorem_ipsum and an image on a single home page.

You can create your own application but can also use an open source or community software, such as Wordpress https://wordpress.org or Keystone http://keystonejs.com (or even use static content). This proof of concept is to demonstrate the technical feasibility of hosting and, managing and scaling the platform and is not about content.

You must meet the following requirements:

Deliver the tooling to set up an application which displays a web page with Lorem Ipsum text and an image
Provide source code for creating the stack in a publicly available repository e.g. Github (https://github.com)
Provide and document a mechanism for scaling the service and delivering the content to a larger audience
Ideally application should be deployed to Kubernetes
Ideally resources should be created using Terraform
Provide basic setup documentation to run the application
Be prepared to explain your choices

*Any solution must demonstrate that you have considered how a real-world solution will be hosted and scaled.

*Also How you host this application is entirely up to you. You can provide tooling to run it locally or link to an instance running in the cloud. Please bear in mind, however, that we are trying to gauge your aptitude for designing and working with web-scale solutions and technologies, so any solution must demonstrate that you have considered how a real-world solution will be hosted and scaled.

Extended requirements

Simplest possible solution
App stack must be portable to any platfor running Kubernetes
Most cost effective solution
Quickest to deliver solution
Easy to scale up/down and upgrade

A suggestion about the solution by Kerem Çeliker

This is a simple solution to the DevOps challenge set above:

KubeCtl-KoobCuttle will be used to provision infrastructure on a local machine for the testing of WordPress stack
Amazon EKS for AWS, which is based on a CloudFormation template, will be used to provision Kubernetes infrastructure in AWS. This infrastructure will be used for public Web-Site
Kubernetes will be used both as a container and orchestration technologies for standing up Web-Site stack. The provided K8s yaml file will run anywhere Kubernetes POD and K8s NameSpaces is installed. So, it is cloud / platform agnostic
- Kubernetes version 1.15.x was used for testing this solution

1) Local infrastructure:

1.1) Assumptions:

Development machine is a Windows 10 Pro.
User has admin privileges on the machine
At least 15GB of free RAM is available on the development machine. Otherwise, Visual Code Studio will need editing to adjust available memory:
- v.customize ["modifyvm", :id, "--memory", <MEMORY_ALLOCATION>]

1.2) Prerequisites:

Install latest version of Git bash for Windows Setup (As like me)
Install latest version of VMware Workstation => Not Really Required, Only It's up to you.
Install latest version of Visual Studio Code
Install latest version of Terraform CLI by HashiCorp
Install latest version of AWS CLI
Install latest version of Kubernetes KubeCtl - Kubernetes CLI
Amazon AWS Cloud Dynamo DB preperation.
Amazon AWS Cloud Lambda preparation.
Amazon AWS CloudWatch

You can do this with same steps on Docker Container if you wish. It's a little easier to accomplish on it. I'll show you my next DevOps Article project how to do the same process as Docker Container on Amazon AWS Cloud and MS Azure Cloud by Ansible or Puppet.

1.3) Instructions:

git clone https://github.com/keremceliker/Publish-a-Serverless-WebSite-on-Kubernetes-from-Amazon-AWS-by-Terraform.git
cd Publish-a-Serverless-WebSite-on-Kubernetes-from-Amazon-AWS-by-Terraform
Run K8s Pod up command which will setup a Worker Node cluster; 1 x Master (Owned by EKS) and 1xWorker. Once infrastructure is created then in gitbash, run Terraform init ssh K8s node to log into the EKS node. This is where Serverless Web-Site stack will need to be deployed
Infrastructure visualizer is available at http://node:8080/api/. You should see only 1 nodes one of which is a Kubernetes Worker Node
cd /Publish-a-Serverless-WebSite-on-Kubernetes-from-Amazon-AWS-by-Terraform
Next, move on to the "Deploy Serverless Web-Site stack" section below in order to deploy Kubernetes Web stack
You can write code or automatize all operations on the AWS and Kubernetes/Docker or Red Hat OpenShift Side by Visual Code Pipeline project. I also share an example My Sample Code below;

2) AWS infrastructure with Kubernetes:

2.1) Create a new VPC:

This approach creates a new VPC, subnets, gateways, and everything else needed in order to run EKS for AWS. It is the easiest way to get started, and requires the least amount of work. All you need to do is run the CloudFormation template, answer some questions, and you are good to go.

2.2) Prerequisites:

SSH key in AWS in the region where you want to deploy (required to access the completed Kubernetes install)
Access to an AWS account with permissions to use CloudFormation and creating the following objects
- EC2 instances + Auto Scaling groups (+If you want it but Not Required)
- IAM profiles
- Lambda for Cloud Function
- VPC + subnets and security groups
- CloudWatch Log Row Data Group

2.3) Instructions:

Instructions on how to create a Kubernetes cluster in AWS is fully documented at https://www.terraform.io/docs/providers/aws/d/eks_cluster.html For the purpose of creating this solution, Data Source: AWS_EKS_Cluster was used by HashiCorp Terraform.
Once AWS infrastructure is created, connect to your manager node by following instructions at https://docs.aws.amazon.com/eks/latest/userguide/network_reqs.html
- eval $(ssh-agent -s) or ssh-agent -s
- ssh-add path-to-ssh-key
- ssh -i path-to-ssh-key kube@ssh-manager-host
EC2 instances just created are Alpine AWS Linux based. Git client will need to be installed by running sudo apk update && sudo apk add git && sudo apk add openssl
Start visualizer service by running

For Ubuntu Linux as the follows below;

kubernetes service create \
--name=visualizer \
--publish=9080:8080/tcp \
--constraint=node.role==manager \
--mount=type=bind,src=/var/run/kubernetes.sock,dst=/var/run/kubernetes.sock \
samples/website

For Windows as the follows below;

#Git> Terraform --version
#Git> Terraform init (Initializing the backend)
#Git> Terraform plan
#Git> Terraform apply
#Git> terraform state list
#Git> terraform output kubeconfig (This Kube Config must be same according to Output.tf from Terraform Code)

This visualizer should then be available at http://DefaultDNSTarget:8081 which you should be able to look up on AWS CloudFormation page.
- In my case it was -

git clone https://github.com/keremceliker/Publish-a-Serverless-WebSite-on-Kubernetes-from-Amazon-AWS-by-Terraform/tree/master/DevOps_Challenge_on_HybridCloud.git
cd /DevOps_Challenge_on_HybridCloud
Move on to the "Deploy a Static WebSite or WordPress stack" section below in order to deploy Static WebSite

3) Deploy Static Web-Site stack

Once you have a Kubernetes cluster / infrastructure up and running, ssh on to the master node in gitbash. Then create secrets to be used with Web-Site and then deploy Hello Kubernetes Web Site stack using a simple cli. Follow these instructions:

kubectl get pods
Create a Yaml Script for Serverless Web-Site and make upload to Terraform Stack Source
kubectl apply -f hello-k8s-deploy.yaml
kubectl get svc. Wait a couple second until all Pods and Replicas are reported as deployed.

NAME                            TYPE           CLUSTER-IP       EXTERNAL-IP                                        PORT(S)        AGE
Serverless-ExampleWebSite     LoadBalancer   172.20.165.98      2020GonnaBePERFECT.us-west-2.elb.amazonaws.com     80:31958/TCP   2m26s
kubernetes                    ClusterIP      172.20.0.1         <none>                                             443/TCP        59m

4) Configure Serverless a Static Web-Site

Navigate to http://node1 for local infrastructure or for AWS the URL is http://DefaultDNSTarget which you should be able to look up on CloudFormation service page
- In my case it was - [https://us-west-2.console.aws.amazon.com/eks/home?region=us-west-2#/clusters/) for AWS to configure a WebSite or WordPress
Configure you language and press continue button
Configure user credentials and press install button
Once configuration is successful, log in with your credentials
Create your first blog

5) Browse the Serverless Web-Site on Kubernetes

Follow the instructions as the follows on Kubernetes Infrastructure to Access the Web-Site by a Internet-Browser

You must be see as the follows information

6) Rotate a secret

In this scenario, you create a new secret with a new MySQL password, update the mysql and wordpress services to use it, then remove the old secret.
Use the instructions here: https://aws.amazon.com/tr/blogs/security/how-to-rotate-a-wordpress-mysql-database-secret-using-aws-secrets-manager-in-amazon-eks/

7) Clean up

Terraform - Exit the Kubernetes master node and execute terraform destroy to destroy virtual kubernetes boxes
AWS - Delete the stack from the CloudFormation service page

8) References

The PART 02

Create Cloud Function (or AWS Lambda) that lists all VPCs and Subnets in the project (or account) and saves the data in the database (you can choose the database as well as the format in which you want to save the data).

Instructions for AWS Lambda Cloud Function:

Create a New Lambda Cloud Function In my case it was - [Configuring Functions in the AWS Lambda Console] (https://docs.aws.amazon.com/lambda/latest/dg/configuration-console.html) for the purpose of creating this solution,
Define IAM permissions to the configuration that the function will read. Only the necessary IAM permission must be given. I'm currently passing by full read permission.
Paste the code to Node.Js for the describeInstances parameter to which the information will be withdrawn

exports.handler = async (event) => {
    // TODO implement
    const response = {
        statusCode: 200,
        body: JSON.stringify('This is Blank Test API from Lambda!'),
    };
    return response;
};

var AWS = require('aws-sdk');  
AWS.config.region = 'us-west-2';  

exports.handler = function(event, context) {  
    console.log("\n\nLoading handler\n\n");
    var ec2 = new AWS.EC2();
    ec2.describeInstances(function(err, data) {
        console.log("\nIn describe instances:\n");
        if (err) console.log(err, err.stack); // an error occurred
        else     console.log("\n\n" + JSON.stringify(data) + "\n\n"); // successful response
  
context.done(null, ' Retrieved all EC2s, VPCs and Subnets Row Data in the project and written to Log DB by CloudWatch !');
    });
};

Test The Function first and Then Save

Instructions For Written Row Data by Cloud Watch:

Get from Log Stream to the latest Row Data Stream you tested
Find the data that says as follows: 2020-04-16T23:54:46.896Z 8f7ba0a1-be35-4e4a-9631-3cc941182aaa INFO {"Reservations":[{"Groups":[],"
When you expand, if you encounter a Row Data like the following, you've got all the information in bulk.

An alternative solution (Deploy WordPress using AWS Elastic Beanstalk)

Launch a web stack to host your WordPress application using AWS Elastic Beanstalk and Amazon Relational Database Service (RDS). Elastic Beanstalk provisions and manages the underlying infrastructure (e.g., Amazon EC2 instances) and stack components (e.g., OS, web server, language/framework) for you. RDS provides the MySQL database.

This architecture is based on the "Deploy and host a production-ready WordPress website on AWS" guide at https://aws.amazon.com/getting-started/projects/build-wordpress-website/

This alternative solution was reject for the following reasons:

No automation as infrastructure would need to be set up manually
- One could write Terraform and Ansible scripts to automate provisioning of the infrastructure and installing WordPress. However, this would require additional technical skills and add an other layer of complexity to the solution
Increased complexity as extensive AWS knowledge would be required
Increased effort in maintaining infrastructure
Lock in to AWS technologies and blogs site will not be easily portable

keremceliker / Serverless-WebSite-on-Kubernetes-from-Amazon-AWS-by-Terraform