bakhti / stelligent_demo

stelligent_demo

Running the Demo

####Prerequisites:#### Python, pip, boto, and awscli are required to be installed and configured. Once awscli is installed, use aws configure to provide aws access keys.

Install pip:

OSX:

==> sudo easy_install pip

Ubuntu:

==> sudo apt-get install python-pip

Amazon Linux/Redhat/CentOS/Fedora:

==> sudo yum install python-pip

Install packages and configure:

==> sudo pip install boto awscli
==> aws configure

Prepare to launch command:

==> git clone https://github.com/stelligent/stelligent_demo.git
==> cd stelligent_demo

####All-In-One Amazon AWS automated demo from a single cli command:####

==> ./go.py build

Options:

• -l xx.xx.xx.xx yy.yy.yy.yy : list IP's from which to limit access. (Default: open to all)
• --region us-xxxx-# : Build stack is specific region. (Default: us-east-1)

To destroy a stack created by this script run:

==> ./go.py destroy

A list of launched stacks will be displayed from which you can select the one to destroy.

Demo Architecture

• Cloud Deployment with CloudFormation.

• Configuration Management with Puppet.

• Continuous Delivery with Jenkins, CodeDeploy, and Docker.

• High Availability with ELB and ASG.

• S3 for static templates, manifests, encrypted keys, and demo outputs.

• RDS for storing image tags and paths.

• IAM and S3 roles and policies for security.

• ElasticBeanstalk for Docker containers.

• ElastiCache for user session data.

• Testing via ChaosMonkey, ServerSpec, TestKitchen, Cucumber, OWASP ZAP, CloudCheckr, evident.io, and Trusted Advisor.

• Vagrant for Developer Environments.

This demo creates ElasticCloud and VirtualPrivateCloud infrastructure and deploys inside with one command. One pipeline control box with Puppet and Jenkins, a CodeDeploy Nginx/PHP-FPM Application via AutoScalingGroup, a Docker container via ElasticBeanstalk, private subnet Multi-AZ HA RDS databases , and ElastiCache for temporary user session data. ElasticLoadBalancers, resolved by Route53 RecordSets, sit in front of both CodeDeploy and Docker web tiers. SimpleStorageService is used for authenticated retrieval of static templates, manifests, and encrypted keys needed upon resource bootstrap. S3 is also used for the running demo to store logs and other outputs securely.

Pipeline Blueprint (COMPLETE)

• deploy-pipeline-1-command (Bash script launches all jobs and resources)
• provision-environment (CloudFormation)
• node-configuration (Packages, Files, Services)
• poll-version-control (GitHub every minute https://github.com/stelligent/stelligent_demo)
• app-deployment: (CodeDeploy and Docker)
• configure-local-environment-1-command: (Vagrant and Docker)

Pipeline Blueprint (PENDING):

• node-configuration: (data encryption, security hardening, test db and local tests)
• run-application-build: (Rake, Maven, Ant, Grunt)
• store-distros: (Nexus, Artifactory, S3)
• run-unit-tests: (RSpec, JUnit, XUnit)
• run-static-analysis: (CheckStyle, PMD, Sonar, CodeClimate, JSONLint, ValidateTemplate, ratproxy, Foodcritic)
• run-infrastructure-tests: (ServerSpec, Cucumber)

Pipeline Security:

• Jenkins IAM roles (COMPLETE)
• Ensure latest code on deploy (COMPLETE)
• Layer 4: tcp/ip only from known admin ip/subnets (COMPLETE)
• Layer 7: jenkins application security (COMPLETE)
• Trusted Advisor (PENDING)
• Jenkins Iptables Firewall (PENDING)

Application Security:

• Private VPC subnet (COMPLETE)
• Application Pen Testing https://github.com/OWASP (PENDING)
• Manual Curl Testing for HTTP responses (PENDING)
• Instance Intrusion Detection https://github.com/ossec/ossec-hids (PENDING)
• Remote Logging (https://github.com/Graylog2 or S3) (PENDING)
• Instance Iptables Firewall (PENDING)

process_notes

• uploads jenkins templates to S3
• uploads puppet manifests to S3
• creates keypair for use in demo
• launches cfn stack
  • builds vpc and dependancies
  • builds Webserver ASG and ELB with Route53 Alias
  • sets up S3 bucket for all logging and demo output
    • buckey policy restricts access to trusted IPs
    • website indexing is enabled
  • builds Multi-AZ MySQL RDS for storing image tags and paths
  • launches ElasticBeanstalk for Docker
  • bootstraps jenkins server via cloud init and authenticated S3
    • adds git functionality
    • installs and configures jenkins via puppet
    • pulls jenkins job templates from authenticated S3 and creates jenkins jobs
    • jenkins executes based on SCM
      • pushes code and images to staging (pending)
      • application and security acceptance testing (pending)
      • push to production with CodeDeploy
• vagrant for developer environments
  • launches docker and codedeploy web tiers
    • codedeploy php stack mapped to tcp/8010
    • docker python stack mapped to tcp/8011
  • launches jenkins with minimal jobs
    • jenkins gui mapped to tcp/8888
  • launches mysql 5.6 (RDS version)
    • mapped to default tcp/3306

todo

• push private key for jenkins from cfn paramater to s3 kms
• change sleep in cfn-init for instance bootstrap to AWS::CloudFormation::WaitCondition
• rewrite deploy script in python
• ChaosMonkey, ServerSpec, TestKitchen, Cucumber, OWASP ZAP
• ElastiCache for user session data
• replace ElasticBeanstalk with ElasticContainerService in CloudFormation for Docker
• replace xml erb jenkins templates with single seed and dsl job definitions

questions

• Diagram of the stages and activities in the deployment pipeline?
• Deployment time?
• How does Vagrant get launched by the user/developer?
• Use Dashing dashboard to show metrics? (Jonny has a CFN template)?
• Make CloudFormation work in multiple regions and AZs
• Consider using m3 default instance types
• Are you doing any encryption (in transit or at rest?)?
• Purpose of restricting IP endpoints in command?