virtuman / StakaterPlatform

Stakater platform provides out of the box stacks to control, monitor, log, trace and security for applications deployed on kubernetes using CI/CD pipelines.

StakaterPlatform

Problem Statement

When an individual or a company deploys a kubernetes cluster for workload management. What is the next thing required. Security, Monitoring, Logging, Tracing, Alerting and CI/CD for workloads, a lot of great opensource tools exist for these processes and selecting a tools based on industry's standards and recommendations is quite a hard & duanting task as it requires a lot of research that fulfill the requirements.

Solution

We at Stakater understand the industry's problem and keeping that in mind we have developed a consolidated solution named StakaterPlatform that will help the community to adopt Security, Monitoring, Logging, Tracing, Alerting and CI/CD processes for thier workloads.

Introduction

StakaterPlatform gives a head-start to individuals and companies with a set of seven stacks containing best of the breed opensource tools based on industry's recommendation and best practices for monitoring, logging, security, tracing and alerting of Kubernetes cluster. End-users only need to focus on developing their application while everything else will be managed by StakaterPlatform!

StakaterPlatform uses community managed helm charts and docker images.

For detailed walk-through please visit StakaterPlatform on Stakater Playbook

StakaterPlatform consist of 7 stacks:

• Control
• Delivery
• Logging
• Monitoring
• Security
• Alerting
• Tracing

Table for Stacks tools:

Control	Delivery	Logging	Monitoring	Security	Alerting	Tracing
External DNS	Jenkins	Elasticsearch	Prometheus Operator	Keycloak	Uptime Robot	Istio
Nginx Ingress	Sonatype-Nexus	Fluentd	Metrics Server	Proxy Injector	Slack
Descheduler	RDLM	Eventrouter
Forecastle		Kibana
Reloader		Cerebro
Sealed Secret Controller		Elasticsearch Curator
K8S Dashboard		Logrotate
Gitwebhookproxy		Konfigurator
Ingress Monitor Controller
Xposer

Prerequisites

• Kubernetes cluster with at least 8 VCPUS & 32 GB of RAM
• A working domain (e.g. stakaterplatform.com )
• SSL Certificate for that domain. Creating & using custom SSL certificates

Tools/Terms

• Flux: You define the entire desired state of your cluster in git and flux ensures that the current state matches the one declared in repo.
• Kubeseal: Required to encrypt base64 encoded secrets to commit-able(git) sealed-secrets

Install from local machine

Video Tutorial

Installation tutorial for local machine can be seen here

Prerequisites

• kubectl (between v1.11 & v1.15.3)
• helm (v2.15.0 or lower)
• kubeseal

1. Duplicate this repository.

2. Update configuration variables in variables.config file and provide the relevant values.

3. [Recommended but optional] To take full advantage of the tool stack configure Additional Variables as well.

4. Ensure that correct context is set for kubectl & helm.

5. run make configure, this will make all required substitutions based on configuration variables in the repository. When prompted commit those changes, don't commit them if you want to run everything from your local machine i.e. No flux hence no GitOps.

6. For deployment there are two options:

   a. Using flux, Add the public SSH key of flux(configs/flux.pub) to your Git repository with write access. For flux run make deploy-flux and provide git credentials on prompt.

   b. Without flux, just don't add flux key and this will remove the process of GitOps(using flux) from your cluster. It removes the need of having your repository available remotely. Run make deploy

7. Estimated time for everything to be up and running is 5-10. Use the printed dashboard token to access the Kubernetes dashboard at dashboard-control.YOURDOMAINNAME

8. Visit https://forecastle-control.YOURDOMAINNAME and you'll be able to view all applications deployed by StakaterPlatform.

Note: Since variables.config file and configs/ directory contains private information those files are not being tracked in git and won't/shouldn't be committed. In case you want to commit those changes run make track-secrets.

Install via GitLab CI Pipeline

1. Duplicate this repository in a GitLab account.
2. Update configuration variables in variables.config file and provide the relevant values.
3. Create a Personal Access Token on GitLab and note down Personal Access Token (Available only when created)
4. Configure CI/CD Environment variables in <Your GitLab Project> -> Settings -> CI/CD.
5. Add Following variables for the pipeline

Pipeline Variables	Description
KUBE_CONFIG	Base64 encoded KubeConfig of the kubernetes cluster you want to deploy on
REPO_ACCESS_TOKEN	Personal access token generated in Step 3
TARGET	Make file target. Value: pipeline-deploy
AWS_ACCESS_KEY_ID	(Define only if CLOUD_PROVIDER is aws) AWS Access Key Id.
AWS_SECRET_ACCESS_KEY	(Define only if CLOUD_PROVIDER is aws) AWS Secret Access Key.

6. Add the public SSH key(Deploy Keys) to your GitLab account with write access printed at the end of pipeline logs.
7. Once key is added StakaterPlatform will deploy on your cluster (Estimated time: 5-10 minutes). 🎊
8. Use the printed dashboard token to access the Kubernetes dashboard at dashboard-control.DOMAIN

Flow Diagram for Deployment via GitLab CI Pipeline

Verification

Locally

Run make verify to run tests to ensure that all the relevant endpoints are up and running.

GitLab CI

Run pipeline with Pipeline variable: TARGET = verify

StakaterPlatform offers a tool Forecastle to access tools from a single panel. Go to forecastle-control.DOMAIN to access forecastle after complete deployment where you can access all the tools offered by StakaterPlatform and verify if working fine

Default Credentials

See default username and passwords set for tools here

Basic Configuration

Variables	Description	Default
CLOUD_PROVIDER	Name of the k8s cloud provider	nil (aws | azure)
DNS_PROVIDER	Cloud DNS Provider	aws (Route53)
EXTERNAL_DNS_AWS_ACCESS_KEY_ID	AWS Access Key Id having access to create/delete/update Route53 HostedZone entries	nil
EXTERNAL_DNS_AWS_SECRET_ACCESS_KEY	AWS Secret Access Key having access to create/delete/update Route53 HostedZone entries	nil
DOMAIN	Domain to use for StakaterPlatform	nil
BASE64_ENCODED_SSL_CERTIFICATE_CA_CRT	Base64 encoded Intermediate Certificate value	nil
BASE64_ENCODED_SSL_CERTIFICATE_TLS_CRT	Base64 encoded Server Certificate value	nil
BASE64_ENCODED_SSL_CERTIFICATE_TLS_KEY	Base64 encoded Certificate Key value	nil
STAKATER_PLATFORM_SSH_GIT_URL	SSH URL for your Github repo.	nil (e.g [e-mail]/stakater/StakaterPlatform.git. Notice : is replaced with / in the URL )
STAKATER_PLATFORM_BRANCH	Branch to use for STAKATER_PLATFORM_SSH_GIT_URL	master

For generating certificates view: Creating & using custom SSL certificates

Detailed Configuration and Stack definitions

See Detailed Configurations for configuring available tools in the stacks.

See Stakater Playbook for stack definitions and configuration options.

Uninstall

Run make destroy to remove StakaterPlatform from your cluster.

Production Configuration and Hardening

• Replace all secrets with sealed-secrets. Stakater Workshop - Sealed Secrets In coming updates for StakaterPlatform this will be followed by default
• Change default usernames and passwords for all tools (variables.config)
• Add your own SSH keys for flux
• Use Identity Providers (e.g. Google, Active Directory etc.) and configure keyCloak to use that. KeyCloak with Identity Providers
• Use keycloak for SSO over all exposed applications
• Enable mTLS between microservices to prevent unauthorized connections between pods
• Only allow restricted access to users other than the administrator for dashboard
• Don't share your kubeconfig file and other secrets, it's suggested to keep such things in vault

Compatibility Matrix

StakaterPlatform has been tested on following kubernetes flavors:

Platform Version	K8s Version	Infrastructure
v0.0.1	1.14	eks.6
v0.0.1	1.14.8	aks

Community

If you have questions, check the Documentation and talk to us on slack #community on Stakater Slack.

Click here to join Stakater on Slack.