Giters

aramallo / augoor-installation

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

augoor-client-installation

Scripts to config and install Augoor System in clients, The objetive is provide installation over Kubernetes for AWS, Azure, and GCP Clouds

Overview of Augoor

TL;DR

  1. Create corresponding infrastructure as per provider
  2. Install RabbitMQ
  3. Create a file with all of your custom values in examples/my-values.yaml
  4. Install the Helm Chart as follows
cd helm
helm install my-release augoor . -f ../examples/my-values.yaml --namespace [augoor-k8s-namespace | augoor-openshift-project]

Introduction

The installation does have 3 main stages:

  • Infrastructure Prerequisites
  • RabbitMQ installation
  • Service deployment
  • internet access from the cluster to download models from models-sftp.augoor.com

Infrastructure Prerequisites

Kubernetes or OpenShift Cluster

Storage

  • An NFS mount point in the following services:

    Provider Service Gb/s Config
    AWS EFS 0.5 General Purpose/Elastic Throuput
    Azure Storage Account 0.5 File Storage/File Share NFS
    GCP Filestore instance 0.5

Database

  • A PostgreSQL database in the following services:

    Provider Service Type
    AWS RDS db.t3.micro
    Azure Azure Database Standard_B1ms
    GCP Cloud SQL db-g1-small

  • Database and user needed!!

Others

Configuration steps as per provider

IMPORTANT: This is a suggested infrastructure to create a PoC only, please check with your infrastructure team for Production environment

RabbitMQ installation

Please follow RabbitMQ installation guide before to continue

Service Deployment

Service deployment is being done by Augoor Helm Chart. This chart bootstrap an Augoor deployment on a Kubernetes cluster using the Helm package manager. This chart bootstrap an Augoor deployment on a OpenShift cluster using the Helm package manager.

RabbitMQ Check

  1. Check Install RabbitMQ in the corresponding configuration for the provider

  2. Make sure the following Secrets have been created in RABBIT_MQ Namespace:

    • rabbitmq-ca
    • tls-secret

  3. Make sure the following Config Maps have been created in Augoor Namespace:

    • cacerts
    • server.crt

Configure the required configuration for the new installation

  1. The Parameters section lists the parameters that should be configured during installation. In the examples/ directory you can find some common examples
  2. After the creation of examples/my-values.yaml Augoor is ready to be install as follows:

Installing the Chart

To install the chart with the release name my-release:

cd helm
helm install my-release augoor -f ../examples/my-values.yaml --namespace [augoor-k8s-namespace | augoor-openshift-project]

Uninstalling the Chart

To uninstall/delete the my-release deployment:

helm delete my-release

The command removes all the Kubernetes components associated with the chart and deletes the release.

Parameters

Global parameters

Name Description Value
global.JWT_PRIVATE_KEY JSON WEB Tokens private key. See JWT Configurarion section ""
global.JWT_PUBLIC_KEY JSON WEB Tokens public key. See JWT Configurarion section ""
global.appurl Application URL's full domain only. i.e. auggor.my-domain.com ""
global.cipherPassphrase Some passphrase for encrypt messages between Auth and CodeProcessor ""
global.certificateArn Only for AWS. A valid ARN certificate for the ingress ALB ""
global.googleCertificateName Only for GCP. A valid certificate Name for the ingress ALB ""
global.googleStaticIPName Only for GCP. A valid reserverd IP Name for the ingress ALB ""
global.cloudProvider Cloud Provider. The value can be either of: aws, azure, gcp or minikube ""
global.containerRegistry Global Docker image registry ""
global.containerRepository Global Docker image repository, e.g: augoor.azurecr.io/$containerRepository/image:tag "serenity"
global.customDomainProvider Needed if the Provider does have a custom domain instead of [github/gitlab/etc].com ""
global.environment Environment for the deployment prod
global.nodeGroupCPU Kubernetes Node group for general services. Not needed for OpenShift ""
global.nodeGroupGPU Kubernetes Node group with GPU. Not needed for OpenShift ""
global.postgresqlServer PostgreSQL server to connect to ""
global.postgresqlPort Port to connect to PostgreSQL server 5432
global.queuePass Password for the user above. Check augoor-user-credentials Secret in RabbitMQ Namespace ""
global.queuePort Port to connect to RabbitMQ service 5671
global.queueServer Deployed RabbitMQ service. eg: rabbitmq-cluster.rabbit-namespace.svc ""
global.queueTLS Whether to use TLS to encrypt RabbitMQ or not true
global.queueTLSstring Whether to use TLS to encrypt RabbitMQ or not using as values "0" or "1" "1"
global.queueUser User for RabbitMQ. Generated in the Configuration Steps ""
global.sonarUrl URL used for SonarQube connection "http://sonarqube:9000"
global.sonarPassword Password for SonarQube Admin user. Sonar Default will be changed during helm install ""
global.sonarUser Admin user for SonarQube admin
global.vaultEnabled Enable Vault false
global.volumeRootPath NFS Root Path for mounting the directories in Augoor container /augoor
global.volumeServer NFS server to connect to ""
global.ScmDomainProvider Pair values separated by :, where the first value is the repo provider and the second one is the hostname of the repo provider. It can be more than one pair defined, just separate it with a comma ,. Repo Provider values supported: "github", "gitlab", "azure", "bitbucket". Some examples: "github:github.com", "github:my-internal-repos.com", "bitbucket:bitbucket.org,gitlab:gitlab.com" ""
global.sftpServer SFTP server url to connect to (needed for download model binaries for inference) ""
global.sftpUser SFTP user for connect to server ""
global.sshKeyPath Path of SSH key needed for connect to SFTP server ""

SharedIngres parameters (Only for AWS)

Name Description Value
sharedIngress.securityGroups comma separated list of AWS security Groups for the ALB. eg: [sg-####] []
sharedIngress.subnets comma separated list of AWS subnets id for the ALB. eg: [subnet-####,subnet-####] []

Auth parameters

Required Parameters:

Name Description Value
auth.flywayUser PostgreSQL DB admin user. ""
auth.flywayPassword Password for the PostgreSQL DB admin user ""
auth.postgresqlPassword Password for augooruser PostgreSQL user ""
auth.reposProvider List of repositories to be used by Augoor. See reposProvider section []

Optional Parameters, if set, will override the corresponding global.value:

Name Description Value
auth.enabled Set to False for not deploying Auth service True
auth.environment Environment for the deployment ``
auth.containerRegistry Global Docker image registry ``
auth.postgresqlServer PostgreSQL server to connect to ``
auth.postgresqlPort Port to connect to PostgreSQL server ``

Codeprocessor parameters

Required Parameters:

Name Description Value
codeprocessor.foreFrontServer ForeFront URL service ""
codeprocessor.foreFrontApiKey ForeFront API Key ""

Optional Parameters, if set, will override the corresponding global.value:

Name Description Value
codeprocessor.enabled Set to False for not deploying Code Processor service True
codeprocessor.containerRegistry Global Docker image registry ``
codeprocessor.volumeServer NFS server to connect to ``
codeprocessor.volumeRootPath NFS Root Path for mounting the directories in codeprocessor container ``

Context parameters

Optional Parameters, if set, will override the corresponding global.value:

Name Description Value
context.enabled Set to False for not deploying Context service True
context.containerRegistry Global Docker image registry ``
context.volumeServer NFS server to connect to ``
context.volumeRootPath NFS Root Path for mounting the directories in context container ``

Context-assistant parameters

Optional Parameters, if set, will override the corresponding global.value:

Name Description Value
contextassistant.enabled Set to False for not deploying Context service True
contextassistant.openAIKey Open AI API KEY is needed for the context-assistant to work ""

Forefront Proxy parameters

Optional Parameters, if set, will override the corresponding global.value:

Name Description Value
forefrontproxy.enabled Set to False for not deploying Index service True
forefrontproxy.containerRegistry Global Docker image registry ``

Indexer parameters

Optional Parameters, if set, will override the corresponding global.value:

Name Description Value
indexer.enabled Set to False for not deploying Index service True
indexer.containerRegistry Global Docker image registry ``
indexer.volumeServer NFS server to connect to ``
indexer.volumeRootPath NFS Root Path for mounting the directories in indexer container ``

Job Starter parameters

Optional Parameters, if set, will override the corresponding global.value:

Name Description Value
jobstarter.enabled Set to False for not deploying Index service True
jobstarter.containerRegistry Global Docker image registry ``
jobstarter.maxReplicas Maximum quantity of GPU instances to be created. Please check your GPU Node group configuration "1"
jobstarter.sleepBetweenEvaluation Seconds the job-starte will wait before getting a new message count of the Queue "30"
jobstarter.serviceTriggerThershold Value limit to reach to trigger a model-inference job creation, with AVG metric: 1 or more messages will trigger a inference job creation "0"
jobstarter.metric Metric used for calculate if a new job need to be created can be "AVG","SPEED","MIN","MAX" "AVG"

Model Inference parameters

Optional Parameters, if set, will override the corresponding global.value:

Name Description Value
modelinference.enabled Set to False for not deploying Inference service True
modelinference.containerRegistry Global Docker image registry ``
modelinference.environment Environment for the deployment ``
modelinference.inactivityTimeout Seconds the application will wait for new message in the queue before shutdown pod 300

Search parameters

Optional Parameters, if set, will override the corresponding global.value:

Name Description Value
search.enabled Set to False for not deploying Search service True
search.containerRegistry Global Docker image registry ``
search.volumeServer NFS server to connect to ``
search.volumeRootPath NFS Root Path for mounting the directories in search container ``

Semantic Conversation parameters

Optional Parameters, if set, will override the corresponding global.value:

Name Description Value
semanticonversation.enabled Set to False for not deploying Semantic Conversation service True
semanticconversation.containerRegistry Global Docker image registry ``
search.volumeServer NFS server to connect to ``
search.volumeRootPath NFS Root Path for mounting the directories in search container ``

SonarQube parameters

Required Parameters:

Name Description Value
sonarqube.postgresqlUser User for sonarqube database ""
sonarqube.postgresqlPassword Password for above user ""

Optional Parameters, if set, will override the corresponding global.value:

Name Description Value
sonarqube.enabled Set to True for deploying SonarQube Community service False
sonarqube.environment Environment for the deployment ``
sonarqube.containerRegistry Global Docker image registry ``
sonarqube.postgresqlServer PostgreSQL server to connect to ``
sonarqube.postgresqlPort Port to connect to PostgreSQL server ``

Spider parameters

Environmental variables description:

Name Description Value
spider.sonarMaxRetry Max amount of retries to connect into Sonarqube API. "30"
spider.sonarRetrySleepInterval Seconds to wait between every retry to connect to Sonarqube API after an analysis. "30"
spider.snQueueContext Queue name for spider publisher. "sn.spider"
spider.rabbitQueue RabbitMQ queue name for spider consumer. "sn.context"
spider.experimental Activate or deactivate experimental features such as assistantdatabase enqueue message. "0"
spider.reconnectionDelay Retries amount to reconnect to RabbitMQ queue. "5"
spider.csaStrategy Enable the strategy to be used on health/code smells metadata generation. Supported values: "SONAR_COMMUNITY","SONAR_ENTERPRISE", "PMD", "COBOL", "DISABLED". Concatenate with a comma , to activate more than one ("PMD,COBOL"). Use "DISABLED" to deactivate step. ""
spider.dependencyGraph Enable the strategy to be used on dependencies metadata generation. Supported values: "EMERGE" , "COBOL" , "DISABLED". Concatenate with a comma , to activate more than one ("EMERGE,COBOL"). Use "DISABLED" to deactivate step. ""

Optional Parameters, if set, will override the corresponding global.value:

Name Description Value
spider.enabled Set to False for not deploying Spider service True
spider.containerRegistry Global Docker image registry ``
spider.volumeServer NFS server to connect to ``
spider.volumeRootPath NFS Root Path for mounting the directories in spider container ``
spider.bitbucketToken BitBucket personal token. Only used if Auth failed to get one ``
spider.azureToken BitBucket personal token. Only used if Auth failed to get one ``
spider.githubComToken BitBucket personal token. Only used if Auth failed to get one ``
spider.githubCorpToken BitBucket personal token. Only used if Auth failed to get one ``

Structure Indexer parameters

Optional Parameters, if set, will override the corresponding global.value:

Name Description Value
structureindexer.enabled Set to False for not deploying Structure Indexer service True
structureindexer.containerRegistry Global Docker image registry ``

Structure Retrieval parameters

Optional Parameters, if set, will override the corresponding global.value:

Name Description Value
structureretrieval.enabled Set to False for not deploying Structure Retrieval service True
structureretrieval.containerRegistry Global Docker image registry ``

UI parameters

Required Parameters IMPORTANT: For OpenShift Only:

Name Description Value
ui.useServiceAccount It will use the already generated augoor-ui-sa with root privileges true

Optional Parameters, if set, will override the corresponding global.value:

Name Description Value
ui.enabled Set to False for not deploying UI service True
ui.containerRegistry Global Docker image registry ``

Assistant Database parameters

Required Parameters

Name Description Value
assistantdatabase.reconnectionDelay Retries to re-connect to queue "5"
assistantdatabase.rabbitQueue RabbitMQ queue name ""
assistantdatabase.getCommitDays Amount of days to consider to extract Git metadata "2000"

Optional Parameters, if set, will override the corresponding global.value:

Name Description Value
assistantdatabase.enabled Set to False for not deploying assistantdatabase service True
assistantdatabase.containerRegistry Global Docker image registry ``
assistantdatabase.volumeServer NFS server to connect to ``
assistantdatabase.volumeRootPath NFS Root Path for mounting the directories in assistantdatabase container ``

JWT Configuration

JSON Web Token (JWT) is an open standard RFC 7519 that defines a compact and self-contained way for securely transmitting information between parties as a JSON object.

Using ES256 algorithm

openssl ecparam -name secp256r1 -genkey -noout -out /tmp/my.key.pem

# Extracts PKCS8 private key
global.JWT_PRIVATE_KEY=`openssl pkcs8 -topk8 -nocrypt -in /tmp/my.key.pem`

# Extracts public key
global.JWT_PUBLIC_KEY=`openssl ec -in /tmp/my.key.pem -pubout`

rm /tmp/my.key.pem

Repos Provider

Here you can have a list of repositories and their OAuth 2.0 attributes. for instance:

if you want to add a GitHub repository:

documentation about how to create Auth App in GitHub here

  reposProviders:
    - name: github
      url: "" # THIS FIELD NEEDS TO BE EMPTY
      admins: user1,user2
#     defaultRole: only if admins is not present can be used to make every user by default admin with the value ROLE_ADMIN
      clientId: Client ID from OAuth Apps
      clientSecret: Client Secret from OAuth Apps
      userAttr: login
      clientName: GitHub
      scope: read:user,repo
      type: GITHUB

if you want to add an Azure repository:

documentation about how to create OAuth 2 web Application in Azure DevOps here

  reposProviders:
    - name: azure
      url: https://dev.azure.com/augoor
      admins: username1,username2
#     defaultRole: only if admins is not present can be used to make every user by default admin with the value ROLE_ADMIN      
      clientId: APP ID from Authorizations
      clientSecret: Client Secret from Authorizations
      userAttr: login
      clientName: Azure repos
      scope: vso.code,vso.project
      type: AZURE_REPOS

And, of course, both repositories

  reposProviders:
    - name: github
      url: "" # THIS FIELD NEEDS TO BE EMPTY
      admins: user1,user2
      clientId: Client ID from OAuth Apps
      clientSecret: Secret from OAuth Apps
      userAttr: login
      clientName: GitHub
      scope: read:user,repo
      type: GITHUB
    - name: azure
      url: https://dev.azure.com/augoor
      admins: username1,username2
      clientId: APP ID from Authorizations
      clientSecret: Client Secret from Authorizations
      userAttr: login
      clientName: Azure repos
      scope: vso.code,vso.project
      type: AZURE_REPOS

if you want to add a GitLab SaaS repository:

documentation to create user-owned application in gitlab here

  reposProviders:
    - name: gitlab
      url: "https://bitbucket.org" 
      admins: user1,user2
      clientId: Application ID from Applications
      clientSecret: Client Secret from Secret field
      userAttr: username
      clientName: GitLab
      scope: read_user,read_api,read_repository
      type: GITLAB

if you want to add a GitLab self-managed repository:

  reposProviders:
    - name: gitlab
      type: GITLAB
      url: "url of your installation of gitlab"
      admins: user1,user2
      clientId: Application ID from Applications
      clientSecret: Client Secret from Secret field
      userAttr: username
      clientName: GitLab
      scope: read_user,read_api,read_repository

About

Languages

Language:Shell 100.0%