Self Managed OSDU

This is a prototype used to evaluate the possibility of providing an easy self managed instance of OSDU on Azure to ease the development experience of developing components or extensions that might run on top or inside of the OSDU Platform that runs in Azure.

Goals:

Build an automation layer on top of OSDU on Azure without impacting the official OSDU Platform.
Ease the amount of required knowledge needed to deploy the platform.
Allow for customizations and configuration.
Provide access to and control of all software components installed.

Architecture:

The Architecture is based on the OSDU on Azure Reference Architecture but an additional diagram is provided to help visualize the architecture here.

Initialize the project

The solution is designed to operate from a fork instance of this project.

Fork the Project into your own GitHub Organization.
Edit the Configuration Settings. (Configuration)
Add any customizations (Optional) (Customizations)

Configure GitHub Secrets

Secrets are managed using Github Repository Secrets some secrets are required to be created manually while others are created automatically by running Github Actions.

Manually Created Secrets

GH_REPO_TOKEN: A personal access token with repo scope.
AZURE_LOCATION: The Azure Region to deploy the resources to.
AZURE_CREDENTIALS: The json output of a Service Principal with Owner Subscription Scope.

SUBSCRIPTION_ID=$(az account show --query id --output tsv)
AZURE_CREDENTIALS="self-managed-osdu-azure-credentials-$(az account show --query user.name -otsv | awk -F "@" '{print $1}')"

az ad sp create-for-rbac --name $AZURE_CREDENTIALS \
  --role "Owner" \
  --scopes /subscriptions/$SUBSCRIPTION_ID \
  --sdk-auth \
  -ojson

# Sample Format
{
  "clientId": "00000000-0000-0000-0000-000000000000",                       # Client ID GUID
  "clientSecret": "**********************************",                     # Client Secret
  "subscriptionId": "00000000-0000-0000-0000-000000000000",                 # Subscription ID GUID
  "tenantId": "00000000-0000-0000-0000-000000000000",                       # Tenant ID GUID
  "activeDirectoryEndpointUrl": "https://login.microsoftonline.com",
  "resourceManagerEndpointUrl": "https://management.azure.com/",
  "activeDirectoryGraphResourceId": "https://graph.windows.net/",
  "sqlManagementEndpointUrl": "https://management.core.windows.net:8443/",
  "galleryEndpointUrl": "https://gallery.azure.com/",
  "managementEndpointUrl": "https://management.core.windows.net/"
}

OSDU_CREDENTIALS: The json output a Service Principal with Contributor Subscription Scope.

SUBSCRIPTION_ID=$(az account show --query id --output tsv)
OSDU_CREDENTIALS="self-managed-osdu-stamp-credentials-$(az account show --query user.name -otsv | awk -F "@" '{print $1}')"

az ad sp create-for-rbac --name $OSDU_CREDENTIALS \
  --role "Contributor" \
  --scopes /subscriptions/$SUBSCRIPTION_ID \
  --sdk-auth \
  -ojson

# Sample Format
{
  "clientId": "00000000-0000-0000-0000-000000000000",                       # Client ID GUID
  "clientSecret": "**********************************",                     # Client Secret
  "subscriptionId": "00000000-0000-0000-0000-000000000000",                 # Subscription ID GUID
  "tenantId": "00000000-0000-0000-0000-000000000000",                       # Tenant ID GUID
  "activeDirectoryEndpointUrl": "https://login.microsoftonline.com",
  "resourceManagerEndpointUrl": "https://management.azure.com/",
  "activeDirectoryGraphResourceId": "https://graph.windows.net/",
  "sqlManagementEndpointUrl": "https://management.core.windows.net:8443/",
  "galleryEndpointUrl": "https://gallery.azure.com/",
  "managementEndpointUrl": "https://management.core.windows.net/"
}

OSDU_CREDENTIAL_OID: The Object ID of the OSDU_CREDENTIALS Service Principal.

az ad sp list --display-name $OSDU_CREDENTIALS --query [].objectId -otsv

OSDU_APPLICATION: The json output of an Azure AD Application.

OSDU_APPLICATION="self-managed-osdu-stamp-application-$(az account show --query user.name -otsv | awk -F "@" '{print $1}')"

az ad app create --display-name $OSDU_APPLICATION \
  --oauth2-allow-implicit-flow \
  --required-resource-accesses @configuration/manifest.json \
  --query '{appId:appId, displayName:displayName, objectId:objectId}' \
  -ojson


  # Sample Format
{
  "appId": "00000000-0000-0000-0000-000000000000",
  "displayName": "osdu-application",
  "objectId": "00000000-0000-0000-0000-000000000000"
}

Generated Secrets

RAND: Unique Random String - Created by Action `Stamp Initialize'
SSH_PASSPHRASE: AKS SSH Key - Created by Action `Stamp Initialize'
SSH_PRIVATE_KEY: AKS SSH Key - Created by Action `Stamp Initialize'
SSH_PUBLIC_KEY: AKS SSH Key - Created by Action `Stamp Initialize'
BUILDER_GROUP: Resource Group Name - Created by Action `Stamp Builder'

Github Actions

Deployment of a self managed osdu instance is performed by executing github actions to work with a Deployment Stamp. Currently there is only support for the deployment of 1 stamp.

Stamp Initialize: This action initializes any neccesary items in github that are necessary to begin (ie: Randomizer Secrets, ssh keys). (Time: ~30s)
Stamp Builder: This action provisions resources necessary to support a provisioning process (ie: Terraform State and Secret Storage). (Time: ~3m)
Stamp Provision: This action provisions resources for the Deployment Stamp. (Time: ~1h)
Stamp Install: This action initializes the Software Configuration process of the Deployment Stamp and the software deployment occurs after pipeline completion. (Time: ~20m)

Note: Prior to running the Stamp Load due to a recent change in Azure AD Applications the AD Application needs to be approved for access. This can be done by accessing the Login Page. Access the $DNS_HOST/login

Stamp Load: This action initializes the partition and loads the necessary data into the Stamp to allow it to fully function. (ie: Entitlements, Schemas, Workflow) (Time: ~20m)
Stamp Uninstall: This action uninstalls the OSDU Stamp and requires Github Secrets used as Feature Flags to enable the activity.

Secret: UNINSTALL - This secret will trigger the feature flag to perform a software uninstall.
Secret: REMOVE - This secret will trigger the feature flag to delete the entire Stamp. (Locks must be manually removed first)

Provisioned Resources

The following resources are created in Azure for a Deployment Stamp.

Builder

Azure Key Vault (Standard)
Azure Storage Account - Table (StorageV2-LRS)

Control Plane

Log Analytics Workspace - 2 Solutions - Container & KeyVault (Pay-as-you-go)
Application Insights (Pay-as-you-go)
Azure Key Vault (Standard)
Azure Storage Account - Table (StorageV2-LRS)
Azure Cosmos DB Account - Gremlin (4000 Max RU's - Shared)
Azure Container Registry - (Standard)
User Managed Identity

Data Plane

Azure Virtual Network
Azure Storage Account - Qty2 (StorageV2-LRS)
Event Grid System Topic
Application Gateway - (WAFv2 w/ Autoscale (2-10))
Public IP Address
Azure Database for Postgres (General Purpose - 4vCore 5GB)
Azure Cosmos DB Account - Core (4000 Max RU's - Shared)
Azure Cache for Redis (Standard 1Gb)
Azure Kubernetes Service - System NodePool (2 Standard_E4s_v3), User NodePool (5-10 Standard_E4s_v3)
User Managed Identity - Qty3

Data Partition

Service Bus Namespace (Standard)
Event Grid Domain w/ 5 Topics (Basic)
Azure Cosmos DB Account - Core (12000 Max RU's - Shared)
Azure Storage Account - Qty2 (StorageV2-GZRS)

Platform Access

Service Validation

Verify the osdu-azure services have successfully started.

Services can be seen by looking in the Azure Portal at the AKS Workloadsunder the namespace osdu-azure.

To optionally further validate a deployment you will need access to the AKS cluster via command line. Please refer to cli-validation for more information.

API Endpoint Validation

Verify the self-managed-osdu instance is available by accessing the Data Platform Endpoint.

The API endpoint DNS name can be found by looking in the Azure Portal at the DNS of the Public IP Address.

AD Application Configuration

The OSDU Stamp AD Application must have the platform settings configured as a Web Platform.

API Requests

The simpliest way to execute against the Platform is to leverage the HTTP Rest Scripts that make testing and executing API calls easier. These scripts are compatable with the VS Code Extension REST Client.

Rest Client Settings can be set to create environments and saved in VS Code Settings.

Retrieve the Rest Client Environment Settings

For convenience the Rest Client Environment Settings can be retrieved from the builder keyvault in the secret restclient. To access secrets from the portal an access policy must be setup first.

This json snippet can then be placed in the .vscode/settings.json file along with a required AD Application Client Secret that is used to authenticate the Rest Client.

Configure RBAC Access

Add a User and entitle the user to the ops group. This can be done by running the manage-user.http script.

Modify the @member_email value (Line 69)
Execute createUser (Line 78)
Execute assignMemberToOpsGroup (Line 158)

Generate an OpenID Connect Token

An initial refresh_token has to be configured and added to the resetclient_settings file as the initial_token.

Access the $DNS_HOST/login page of the platform and obtain an authorization token. image
Add the authorization token to the auth_token.http file. (Line 7) image
Execute authorize (Line 13)
Add the refresh token to the restclient_settings file.

Execute Check Scripts

Check Core API check_core.http
Check File Upload check_file.http
Check Manifest Ingest check_manifest.http
Check CSV Ingest check_csv.http

danielscholl / self-managed-osdu