camilamacedo86 / mobile-security-service-operator

Mobile Security Service Operator

Project health

Table of Contents

• Overview
• Prerequisites
• Getting Started
  • Cloning the repository
  • Minishift installation and setup
  • Installing
  • Creating/Binding an app
  • Deleting/Unbinding an app
  • Uninstalling
• Configuration and Options
  • Customizing the Service
  • Customizing the Database
  • Changing the operator namespace
  • Managing valid namespaces to apply the MobileSecurityServiceApp
  • Changing the Environment Variables values
  • Monitoring Service (Metrics)
  • Configuring the Backup Service
• Architecture
  • CRD Definitions
  • Resources managed by each CRD Controller
  • Oauth Authentication
  • Design Pattern
• Administration
  • Status Definition per Types
• Development
  • Local Setup
  • Running the operator locally
  • Checking Database
  • Debugging
• Makefile command reference
  • Application Commands
  • Local Development
  • CI
  • Tests
• Publishing images
• Tags Release
• Supportability
• Contributing
• Questions

Overview

An Operator based on the Operator SDK to run the Mobile Security Service protected by OAuth on a OpenShift cluster.

Prerequisites

Install Golang

Ensure the $GOPATH environment variable is set

Install the dep package manager

Install Operator-SDK

Install kubectl

Getting Started

Cloning the repository

By the following commands you will create a local directory and clone this project.

$ git clone [e-mail]:aerogear/mobile-security-service-operator.git $GOPATH/src/github.com/aerogear/mobile-security-service-operator

Minishift installation and setup

Install Minishift then enable Operators on it by running the following commands.

# create a new profile to test the operator
$ minishift profile set mss-operator

# enable the admin-user add-on
$ minishift addon enable admin-user

# add insecure registry to download the images from docker
$ minishift config set insecure-registry 172.30.0.0/16

# start the instance
$ minishift start

Installing

Use the following command to install the Operator, Service and its Database.

ℹ️	To install you need be logged in as a user with cluster privileges like the system:admin user. E.g. By using: oc login -u system:admin.

$ make install

To verify that the pods has been successful created you can run the following command.

# Go to the project namespace. E.g mobile-security-service
$ oc project mobile-security-service

# Check the pods which are deployed
$ oc get pods
NAME                                                READY     STATUS    RESTARTS   AGE
mobile-security-service-569d4f8d86-vp5hq            1/1       Running   2          1h
mobile-security-service-db-ccb5856f6-8qnvc          1/1       Running   0          1h
mobile-security-service-operator-785cbdbf46-wq2lf   1/1       Running   0          1h

To verify that the installation was successful completed you can check the AppStatus= status field of Service CR and the DatabaseStatus=OK status field of Database CR in the cluster. For both are expected this status fields be added with the value "OK" when all installation process is finished and are all created successfully as the following example.

# To check the Service CR
$ oc describe mobilesecurityservices
...
  Route Name:                        route
  Size:                              1
Status:
  App Status:       OK
  Config Map Name:  mobile-security-service-config
  Deployment Name:  mobile-security-service
...

# To check the Database CR
$ oc describe mobilesecurityservicedbs
...
  Size:                      1
Status:
  Database Status:  OK
  Deployment Name:  mobile-security-service-db
  Deployment Status:
...

ℹ️	To troubleshooting the installation if it does not complete successfully check the status of its CRs. Note that all objects which are managed by them has an respective status field on it. E.g Deployment Status

Creating/Binding an app

• Create a MobileSecurityServiceApp CR as this example.

• The app name and appId need to be specified into the MobileSecurityServiceApp CR as follows.

    # The appName spec defines the name of the app used to bind the service
    appName: "app"
    # The appId spec defines the appId of the app used to bind the service
    appId: "appid"

• Run the following command to bind the app to the Mobile Security Service

  $ make example-app/apply

ℹ️	This command will execute kubectl apply -f deploy/crds/examples/mobile-security-service_v1alpha1_mobilesecurityserviceapp_cr.yaml and apply the MobileSecurityServiceApp CR example.

Deleting/Unbinding an app

• Run the following command to delete/unbind the app from the service.

  $ make example-app/delete

❗	This command will execute kubectl delete -f deploy/crds/examples/mobile-security-service_v1alpha1_mobilesecurityserviceapp_cr.yaml and delete the MobileSecurityServiceApp CR example.

ℹ️	The Rest Service endpoint to delete it is called in the finalizer of the MobileSecurityServiceApp CR and the CR will be just allowed to be removed when the app is no longer available in its Service.

Uninstalling

Use the following command to delete the Operator, the Service and its Database and all related configuration applied by the install of this project.

$ make uninstall

ℹ️	To uninstall you need be logged in as a user with cluster privileges like the system:admin user. E.g. By using: oc login -u system:admin.

Configuration and Options

Customizing the Service

The Mobile Security Service image and its parameters are configurable and specified by the MobileSecurityService CR.

ℹ️	Default values are applied by the operator in mandatory_specs.go are applied for the mandatory specifications which are not made in the MobileSecurityService CR.

Customizing the Database

The database image and its parameters are configurable and specified by the MobileSecurityServiceDB CR.

ℹ️	Default values are applied by the operator in mandatory_specs.go are applied for the mandatory specifications which are not made in the MobileSecurityServiceDB CR.

Changing the operator namespace

By using the command make install the default namespace mobile-security-service, defined in the Makefile will be created and the operator will be installed in this namespace. You are able to install the operator in another namespace if you wish, however, you need to set up its roles (RBAC) in order to apply them on the namespace where the operator will be installed. The namespace name needs to be changed in the Cluster Role Binding file. Note, that you also need to change the namespace in the Makefile in order to use the command make install for another namespace.

  # Replace this with the namespace where the operator will be deployed.
  namespace: mobile-security-service

Managing valid namespaces to apply the MobileSecurityServiceApp

Only namespaces specified in the environment variable APP_NAMESPACES can be used to apply Apps. If the MobileSecurityServiceApp CR is applied to a namespace that is not specified in APP_NAMESPACES it will be ignored. Refer to configuration in the operator.yaml file.

❗	The values should be split by ;. E.g mobile-security-service-apps;example-namespace-apps

ℹ️	To run the project locally export the ENV VAR. E.g. export APP_NAMESPACES=mobile-security-service-apps

Changing the Environment Variables values

Environment Variables are used to configure the Mobile Security Service Application and Database. For further information on configuration see the Setup and Configuration section.

Monitoring Service (Metrics)

The application-monitoring stack provisioned by the application-monitoring-operator on Integr8ly can be used to gather metrics from this operator and the mobile security service. These metrics can be used by Integr8ly’s application monitoring to generate Prometheus metrics, AlertManager alerts and a Grafana dashboard.

It is required that the integr8ly/Grafana and Prometheus operators are installed. For further detail see integr8ly/application-monitoring-operator.

The following commands will enable the monitoring service where the operator has been installed in the default namespace with the make commands.

make monitoring/install

❗	The namespaces are setup manually in the files ServiceMonitor, Prometheus Rules, Operator Service, and Grafana Dashboard. Following an example from the Prometheus Rules. You should replace them if the operator is not installed in the default namespace.

  expr: |
          (1-absent(kube_pod_status_ready{condition="true", namespace="mobile-security-service"})) or sum(kube_pod_status_ready{condition="true", namespace="mobile-security-service"}) != 3

[source,shell]

ℹ️	The command make monitoring/uninstall will uninstall the Monitor Service.

Configuring the Backup Service

Backup

Install

The backup service is implemented by using integr8ly/backup-container-image. It will do the backup of the database and the APP CRs which are data required to be restore in the case of failures. Following the steps to enable it.

1. Setup the AWS in order to store the backup outside of the cluster. You need to add your AWS details to MobileSecurityServiceBackup as follows or add the name of the secret which has already this data in the cluster.

     # ---------------------------------
     # Stored Host - AWS
     # ----------------------------
   
     awsS3BucketName: "example-awsS3BucketName"
     awsAccessKeyId: "example-awsAccessKeyId"
     awsSecretAccessKey: "example-awsSecretAccessKey"

   ❗	Also, you can add the name of the secret which is created already in the cluster.

2. Run the command make backup/install in the same namespace where the MobileSecurityService Database is installed in order to apply the CronJob which will do this process.

3. Add the label integreatly-middleware-service=true in all namespaces defined in the ENV VAR APP_NAMESPACES in the operator.yaml. (E.g oc label ns/mobile-security-service-apps integreatly-middleware-service=true`). It will backup all MobileSecurityServiceApp CR applied on them.

ℹ️	To install you need be logged in as a user with cluster privileges like the system:admin user. E.g. By using: oc login -u system:admin.

Verifying

To verify that the backup has been successful created you can run the following command in the namespace where the operator is installed.

$ oc get cronjob.batch/mobile-security-service-backup
NAME                             SCHEDULE      SUSPEND   ACTIVE    LAST SCHEDULE   AGE
mobile-security-service-backup   0 * * * *   False     0         13s             12m

To check the jobs executed you can run the command oc get jobs in the namespace where the operator is installed as the following example.

$ oc get jobs
NAME                                        DESIRED   SUCCESSFUL   AGE
mobile-security-service-backup-1561588320   1         0            6m
mobile-security-service-backup-1561588380   1         0            5m
mobile-security-service-backup-1561588440   1         0            4m
mobile-security-service-backup-1561588500   1         0            3m
mobile-security-service-backup-1561588560   1         0            2m
mobile-security-service-backup-1561588620   1         0            1m
mobile-security-service-backup-1561588680   1         0            43s

ℹ️	In the above example the schedule was made to run this job each minute (*/1 * * * *)

To check the logs and troubleshooting you can run the command oc logs $podName -f in the namespace where the operator is installed as the following example.

 $ oc logs job.batch/mobile-security-service-backup-1561589040 -f
dumping mobile_security_service
dumping postgres
==> Component data dump completed
/tmp/intly/archives/mobile-security-service.mobile_security_service-22_46_06.pg_dump.gz
WARNING: mobile-security-service.mobile_security_service-22_46_06.pg_dump.gz: Owner username not known. Storing UID=1001 instead.
upload: '/tmp/intly/archives/mobile-security-service.mobile_security_service-22_46_06.pg_dump.gz' -> 's3://camilabkp/backups/mss/postgres/2019/06/26/mobile-security-service.mobile_security_service-22_46_06.pg_dump.gz'  [1 of 1]
 1213 of 1213   100% in    1s   955.54 B/s  done
ERROR: S3 error: 403 (RequestTimeTooSkewed): The difference between the request time and the current time is too large.

Restore

Following the steps required to be performed in case of be required do the restore based in the backup service.

1. Install the Mobile Security Service by following the steps in Installing.

2. Restore the database with the dump which was stored in the AWS S3 bucket.

   ℹ️	To restore we should run gunzip -c filename.gz | psql dbname

3. It is required re-apply all APP CRs which has an backup in the AWS bucket as well.

Architecture

This operator is cluster-scoped. For further information see the Operator Scope section in the Operator Framework documentation. Also, check its roles in Deploy directory.

ℹ️	The operator, application and database will be installed in the namespace mobile-security-service which will be created by this project.

CRD Definitions

CustomResourceDefinition	Description
MobileSecurityService	Packages, manages, installs and configures the Mobile Security Service on the cluster.
MobileSecurityServiceDB	Packages, manages, installs and configures the Mobile Security Service Database on the cluster.
MobileSecurityServiceApp	Creates and update the app in the REST API.
MobileSecurityServiceBackup	Packages, manages, installs and configures the CronJob to do the backup using the image backup-container-image

Resources managed by each CRD Controller

• Mobile Security Service

  Resource	Description
  configmaps.go	Define the ConfigMap resources required for the Mobile Security Service Application and its Database. It will create the mobile-security-service-app which map the values used in the Environment Variables of both.
  deployments.go	Define the Deployment resource of Mobile Security Service Application, e.g. container and resources definitions.
  route.go	Define the route resource required to expose the Mobile Security Service (REST Service and UI).
  services.go	Define the Service resource of Mobile Security Service Application.

• Mobile Security Service Database

  Resource	Description
  deployments.go	Define the Deployment resource of Mobile Security Service Database. (E.g container and resources definitions)
  pvs.go	Define the PersistentVolumeClaim resource used by its Database.
  services.go	Define the Service resource of Mobile Security Service Database.

• Mobile Security Service Backup

  Resource	Description
  cronjobs.go	Define the CronJob resources in order to do the Backup.
  secrets.go	Define the database and AWS secrets resources created.

Oauth Authentication

An Oauth Proxy container and the required configuration will be setup by default by the operator to provide authentication to the Mobile Security Service.

Design Pattern

The Design Pattern adopted to build the objects in its controllers is Factory Method Pattern.

Administration

Status Definition per Types

• MobileSecurityService

  Status	Description
  appStatus	For this status is expected the value OK which means that all required Kubernetes/OCP objects are created.
  configMapName	Name of the configMap created with the Environment Variables.
  deploymentName	Name of the deployment object created for the App.
  deploymentStatus	Deployment Status from ks8 API (v1beta1.DeploymentStatus).
  serviceName	Name of the service object created for the App.
  serviceStatus	Deployment Status from ks8 API (v1.ServiceStatus).
  routeName	Name of the route object created for the App.
  routeStatus	Route Status from OCP API (v1.Route).

• MobileSecurityServiceDB

  Status	Description
  databaseStatus	For this status is expected the value OK which means that all required Kubernetes/OCP objects are created.
  deploymentName	Name of the deployment object created for the Database.
  deploymentStatus	Deployment Status from ks8 API (v1beta1.DeploymentStatus).
  serviceName	Name of the service object created for the Database.
  serviceStatus	Deployment Status from ks8 API (v1.ServiceStatus).
  PersistentVolumeClaimName	Name of the PersistentVolumeClaimName object created for the Database.

• MobileSecurityServiceApp

  Status	Description
  bindStatus	For this status is expected the value OK which means that the app was created in the API Service.

• MobileSecurityServiceBackup

  Status	Description
  backupStatus	Should show OK when everything is created successfully.
  cronJobName	Name of cronJob resource created by it.
  cronJobStatus	CronJob Status from ks8 API (k8s.io/api/batch/v1beta1/CronJobStatus).
  dbSecretName	Name of database secret resource created in order to allow the integr8ly/backup-container-image connect to the database .
  dbSecretData	Data used into the secret to connect to the database .
  awsSecretName	Name of AWS S3 bucket secret resource used in order to allow the integr8ly/backup-container-image connect to AWS to send the backup .
  awsSecretData	Data used to in the secret to send the backup files to the AWS S3.
  awsSecretDataNamespace	Namespace where the backup image will looking for the of the Aws Secret used.
  encryptionKeySecretName	Name of the EncryptionKey used.
  encryptionKeySecretNamespace	Namespace where the backup image will looking for the of the EncryptionKey used.
  encryptionKeySecretData	Data used into the EncryptionKey.
  hasEncryptionKey	Expected true when it was configured to use an EncryptionKey secret
  databasePodFound	The value expected here is true which shows that the database pod was found.
  servicePodFound	The value expected here is true which shows that the database service was found.

Development

Local Setup

Run the following command to setup this project locally.

$ make setup

Running the operator locally

The following command will install the operator in the cluster and run the changes performed locally without the need to publish a dev tag. In this way, you can verify your code in the development environment.

$ make code/run/local

❗	The local changes are applied when the command operator-sdk up local --namespace=mobile-security-service is executed then it is not a hot deploy and to get the latest changes you need re-run the command.

Checking Database

By the following commands you are able to connect in the Mobile Security Service Database. You can check it by OpenShift UI in the Database’s pod terminal.

# Login into the the Postgres
psql -U postgres

# To connect into the default database
\c mobile_security_service

# To list the tables
\dt

# To select all data from the app table
SELECT * FROM app;

Debugging

Follow the below steps to debug the project in some IDEs.

ℹ️	The code needs to be compiled/built first.

IntelliJ IDEA / GoLand

$ make setup/debug
$ cd cmd/manager/
$ dlv debug --headless --listen=:2345 --api-version=2

Then, debug the project from the IDE by using the default setup of Go Remote option.

Visual Code

$ make setup/debug
$ dlv --listen=:2345 --headless=true --api-version=2 exec ./build/_output/bin/mobile-security-service-operator-local  --

debug the project using the following Visual Code launch config.

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "test",
            "type": "go",
            "request": "launch",
            "mode": "remote",
            "remotePath": "${workspaceFolder}/cmd/manager/main.go",
            "port": 2345,
            "host": "127.0.0.1",
            "program": "${workspaceFolder}",
            "env": {},
            "args": []
        }
    ]
}

ℹ️	The image/tag used from Mobile Security Service is defined in mobile-security-service_v1alpha1_mobilesecurityservice_cr.yaml file.

Makefile command reference

Application Commands

Command	Description
make install	Creates the mobile-security-service namespace, application CRDS, cluster role and service account. Installs the operator and the Service and DB
make uninstall	Uninstalls the operator and the Service and DB. Deletes the mobile-security-service` namespace, application CRDS, cluster role and service account and the app namespace. i.e. all configuration applied by make install
make example-app/apply	Applies the Example App CR (Create/Update app in the cluster and Service.).
make example-app/delete	Deletes the Example App CR (Delete app from the Service).
make refresh-operator-image	Deletes and applies the operator in order to refresh the image when a tag is not changed (development use)
make monitoring/install	Installs Monitoring Service in order to provide metrics
make monitoring/uninstall	Uninstalls Monitoring Service in order to provide metrics, i.e. all configuration applied by make monitoring/install
make backup/install	Installs the backup Service in the operator’s namespace
make backup/uninstall	Uninstalls the backup Service from the operator’s namespace.

Local Development

make code/run-local	Runs the operator locally for development purposes.
make setup/debug	Sets up environment for debugging proposes.
make code/vet	Examines source code and reports suspicious constructs using vet.
make code/fmt	Formats code using gofmt.
make code/gen	It will automatically generated/update the files by using the operator-sdk based on the CR status and spec definitions.

CI

make test/run	Runs test suite
make test/integration-cover	Run coverage check
make image/build/master	Used by CI to build operator image from master branch and add :master tag.
make image/push/master	Used by CI to push the master image to quay.io registry.
make image/build/release	Used by CI to build operator image from a tagged commit and add :<version> and latest tag.
make image/push/release	Used by CI to push the release and latest image to quay.io registry.

Tests

make test/run	Runs test suite
make test/integration-cover	Run coverage check

ℹ️	The Makefile is implemented with tasks which you should use to work with.

Publishing images

Images are automatically built and pushed to our image repository in the following cases:

• For every change merged to master a new image with the master tag is published.

• For every change merged that has a git tag a new image with the <operator-version> and latest tags are published.

If the image does not get built and pushed automatically the job may be re-run manually via the CI dashboard.

Tags Release

Following the steps

• Create a new version tag following the semver, for example 0.1.0

• Bump the version in the version.go file.

• Update the the CHANGELOG.MD with the new release.

• Looking for the SOPs and update the tag for the them in all files (e.g https://github.com/aerogear/mobile-security-service-operator/blob/0.2.0/SOP/SOP-operator.adoc)

• Create a git tag with the version value, for example:

$ git tag -a 0.1.0 -m "version 0.1.0"

• Push the new tag to the upstream repository, this will trigger an automated release by the CI, for example:

$ git push upstream 0.1.0

ℹ️	The image with the tag will be created and pushed to the mobile-security-service image hosting repository by the CI.

⚠️	Do not use letters in the tag such as v. It will not work.

Supportability

This operator was developed using mainly the Kubernetes APIs in order to be compatible with both, however, currently this project requires the usage of the v1.Route to expose the service and OAuth-proxy for authentication which make it unsupportable for Kubernetes.

Contributing

All contributions are hugely appreciated. Please see our Contributing Guide for guidelines on how to open issues and pull requests. Please check out our Code of Conduct too.

Questions

There are a number of ways you can get in in touch with us, please see the AeroGear community.