Istio "Hello World" my way
What is this repo?
This is a really simple application I wrote over holidays a year ago (12/17) that details my experiences and feedback with istio. To be clear, its a really basic NodeJS application that i used here but more importantly, it covers the main sections of Istio that i was seeking to understand better (if even just as a helloworld).
I do know isito has the "bookinfo" application but the best way i understand something is to rewrite sections and only those sections from the ground up.
Istio version used
- 09/22/20: Istio 1.7.2
- 04/28/20: Istio 1.5.2
- 10/12/19: Istio 1.3.2
- 03/10/19: Istio 1.1.0
- 01/09/19: Istio 1.0.5
- Prior Istio Versions
What i tested
- Basic istio Installation on Google Kubernetes Engine
- Grafana, Prometheus, Kiali, Jaeger
- Route Control
- Canary Deployments with VirtualService
- Destination Rules
- Egress Rules
- Egress Gateway
- WebAssembly
- LUA HttpFilter
- Authorization
- JWT Authentication an Authorization
- Service to Service Authentication Policy
- Internal LoadBalancer (GCP)
- Mixer Out of Process Authorization Adapter
- Access GCE MetadataServer
You can also find info about istio+external authorization server here:
What is the app you used?
NodeJS in a Dockerfile...something really minimal. You can find the entire source under the 'nodeapp' folder in this repo.
The endpoints on this app are as such:
/: Does nothing; (source)/varz: Returns all the environment variables on the current Pod (source)/version: Returns just the "process.env.VER" variable that was set on the Deployment (source)/backend: Return the nodename, pod name. Designed to only get called as if the applciation running is abackend(source)/hostz: Does a DNS SRV lookup for thebackendand makes an http call to its/backend, endpoint (source)/requestz: Makes an HTTP fetch for several external URLs (used to show egress rules) (source)/headerz: Displays inbound headers (source)/metadata: Access the GCP MetadataServer using hostname and link-local IP address (source)/remote: Access/backendwhile deployed in a remote istio cluster (source)
I build and uploaded this app to dockerhub at
docker.io/salrashid123/istioinit:1
docker.io/salrashid123/istioinit:2
(basically, they're both the same application but each has an environment variable that signifies which 'verison; they represent. The version information for each image is returned by the /version endpoint)
You're also free to build and push these images directly:
docker build --build-arg VER=1 -t your_dockerhub_id/istioinit:1 .
docker build --build-arg VER=2 -t your_dockerhub_id/istioinit:2 .
docker push your_dockerhub_id/istioinit:1
docker push your_dockerhub_id/istioinit:2
To give you a sense of the differences between a regular GKE specification yaml vs. one modified for istio, you can compare:
- all-istio.yaml vs all-gke.yaml (review Ingress config, etc)
Lets get started
Create a 1.18+ GKE Cluster and Bootstrap Istio
Note, the following cluster is setup with a aliasIPs (--enable-ip-alias )
We will be installing istio with istioctl
gcloud container clusters create cluster-1 --machine-type "n1-standard-2" --zone us-central1-a --num-nodes 4 --enable-ip-alias \
--cluster-version "1.18" -q
gcloud container clusters get-credentials cluster-1 --zone us-central1-a
kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=$(gcloud config get-value core/account)
kubectl create ns istio-system
export ISTIO_VERSION=1.7.2
wget https://github.com/istio/istio/releases/download/$ISTIO_VERSION/istio-$ISTIO_VERSION-linux-amd64.tar.gz
tar xvf istio-$ISTIO_VERSION-linux-amd64.tar.gz
rm istio-$ISTIO_VERSION-linux-amd64.tar.gz
export PATH=`pwd`/istio-$ISTIO_VERSION/bin:$PATH
istioctl install --set profile=demo \
--set values.global.controlPlaneSecurityEnabled=true \
--set meshConfig.enableAutoMtls=true \
--set values.gateways.istio-ingressgateway.runAsRoot=true \
--set meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY \
-f overlay-istio-gateway.yaml
$ istioctl profile dump --config-path components.ingressGateways demo
$ istioctl profile dump --config-path values.gateways.istio-ingressgateway demo
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.7/samples/addons/prometheus.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.7/samples/addons/grafana.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.7/samples/addons/jaeger.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.7/samples/addons/kiali.yaml
kubectl label namespace default istio-injection=enabledIf you see an error: no matches for kind "MonitoringDashboard" in version "monitoring.kiali.io/v1alpha1", run the commands again
Wait maybe 2 to 3 minutes and make sure all the Deployments are live:
- For reference, here are the Istio operator installation options
Make sure the Istio installation is ready
Verify this step by making sure all the Deployments are Available.
$ kubectl get no,po,rc,svc,ing,deployment -n istio-system
NAME STATUS ROLES AGE VERSION
node/gke-cluster-1-default-pool-7907ec5d-4rnc Ready <none> 33m v1.18.6-gke.4801
node/gke-cluster-1-default-pool-7907ec5d-7qsz Ready <none> 33m v1.18.6-gke.4801
node/gke-cluster-1-default-pool-7907ec5d-jrwq Ready <none> 33m v1.18.6-gke.4801
node/gke-cluster-1-default-pool-7907ec5d-nxvx Ready <none> 33m v1.18.6-gke.4801
NAME READY STATUS RESTARTS AGE
pod/grafana-75b5cddb4d-bvwj7 1/1 Running 0 52s
pod/istio-egressgateway-7b49cdb77f-4d2gh 1/1 Running 0 6m21s
pod/istio-ilbgateway-789d466f67-tkqrp 1/1 Running 0 6m21s
pod/istio-ingressgateway-78c9bf4887-4bp2g 1/1 Running 0 6m20s
pod/istiod-95bffc969-qbf8b 1/1 Running 0 6m36s
pod/jaeger-5795c4cf99-2h9sl 1/1 Running 0 50s
pod/kiali-6c49c7d566-bcrdw 1/1 Running 0 48s
pod/prometheus-9d5676d95-t7h7g 2/2 Running 0 54s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/grafana ClusterIP 10.4.12.115 <none> 3000/TCP 52s
service/istio-egressgateway ClusterIP 10.4.14.212 <none> 80/TCP,443/TCP,15443/TCP 6m20s
service/istio-ilbgateway LoadBalancer 10.4.11.160 10.128.0.122 443:31947/TCP 6m20s
service/istio-ingressgateway LoadBalancer 10.4.2.68 34.123.13.130 443:31538/TCP 6m20s
service/istiod ClusterIP 10.4.9.101 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP,853/TCP 6m36s
service/kiali ClusterIP 10.4.5.232 <none> 20001/TCP,9090/TCP 48s
service/prometheus ClusterIP 10.4.7.125 <none> 9090/TCP 54s
service/tracing ClusterIP 10.4.12.189 <none> 80/TCP 50s
service/zipkin ClusterIP 10.4.0.225 <none> 9411/TCP 50s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/grafana 1/1 1 1 52s
deployment.apps/istio-egressgateway 1/1 1 1 6m21s
deployment.apps/istio-ilbgateway 1/1 1 1 6m21s
deployment.apps/istio-ingressgateway 1/1 1 1 6m21s
deployment.apps/istiod 1/1 1 1 6m37s
deployment.apps/jaeger 1/1 1 1 50s
deployment.apps/kiali 1/1 1 1 48s
deployment.apps/prometheus 1/1 1 1 54sMake sure the Istio an IP for the LoadBalancer is assigned:
Run
$ kubectl get svc istio-ingressgateway -n istio-system
export GATEWAY_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo $GATEWAY_IPSetup some tunnels to each of the services
Open up several new shell windows and type in one line into each:
kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=grafana -o jsonpath='{.items[0].metadata.name}') 3000:3000
kubectl port-forward -n istio-system $(kubectl get pod -n istio-system -l app=jaeger -o jsonpath='{.items[0].metadata.name}') 16686:16686
kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=kiali -o jsonpath='{.items[0].metadata.name}') 20001:20001
Open up a browser (4 tabs) and go to:
- Kiali http://localhost:20001/kiali (username: admin, password: admin)
- Grafana http://localhost:3000/dashboard/db/istio-mesh-dashboard
- Jaeger http://localhost:16686
Deploy the sample application
The default all-istio.yaml runs:
- Ingress with SSL
- Deployments:
-
- myapp-v1: 1 replica
-
- myapp-v2: 1 replica
-
- be-v1: 1 replicas
-
- be-v2: 1 replicas
basically, a default frontend-backend scheme with one replicas for each v1 and v2 versions.
Note: the default yaml pulls and run my dockerhub image- feel free to change this if you want.
kubectl apply -f all-istio.yaml
kubectl apply -f istio-lb-certs.yamlNow enable the ingress gateway for both external and internal loadbalancer traffic on only port :443:
kubectl apply -f istio-ingress-gateway.yaml -f istio-ingress-ilbgateway.yaml
kubectl apply -f istio-fev1-bev1.yamlWait until the deployments complete:
$ kubectl get po,deployments,svc,ing
NAME READY STATUS RESTARTS AGE
pod/be-v1-58855cf854-t6rz9 2/2 Running 0 68s
pod/be-v2-8475cfb896-wxt4z 2/2 Running 0 68s
pod/myapp-v1-7fd949f8fb-cbkgp 2/2 Running 0 68s
pod/myapp-v2-65648669b-8q6ng 2/2 Running 0 68s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/be-v1 1/1 1 1 68s
deployment.apps/be-v2 1/1 1 1 68s
deployment.apps/myapp-v1 1/1 1 1 68s
deployment.apps/myapp-v2 1/1 1 1 68s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/be ClusterIP 10.4.0.236 <none> 8080/TCP 68s
service/kubernetes ClusterIP 10.4.0.1 <none> 443/TCP 37m
service/myapp ClusterIP 10.4.10.14 <none> 8080/TCP 69s
Notice that each pod has two containers: one is from isto, the other is the applicaiton itself (this is because we have automatic sidecar injection enabled on the default namespace).
Also note that in all-istio.yaml we did not define an Ingress object though we've defined a TLS secret with a very specific metadata name: istio-ingressgateway-certs. That is a special name for a secret that is used by Istio to setup its own ingress gateway:
Ingress Gateway Secret in 1.0.0+
Note the istio-ingress-gateway secret specifies the Ingress cert to use (the specific metadata name is special and is required)
apiVersion: v1
data:
tls.crt: _redacted_
tls.key: _redacted_
kind: Secret
metadata:
name: istio-ingressgateway-certs
namespace: istio-system
type: kubernetes.io/tlsRemember we've acquired the $GATEWAY_IP earlier:
export GATEWAY_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo $GATEWAY_IPSend Traffic
This section shows basic user->frontend traffic and see the topology and telemetry in the Kiali and Grafana consoles:
Frontend only
So...lets send traffic with the ip to the /versions on the frontend
for i in {1..1000}; do curl -k https://$GATEWAY_IP/version; sleep 1; doneyou may need to restart the ingress pod if the certs they used didn't pickup
INGRESS_POD_NAME=$(kubectl get po -n istio-system | grep ingressgateway\- | awk '{print$1}'); echo ${INGRESS_POD_NAME};
kubectl delete po/$INGRESS_POD_NAME -n istio-systemYou should see a sequence of 1's indicating the version of the frontend you just hit
111111111111111111111111111111111
(source: /version endpoint)
You should also see on kiali just traffic from ingress -> fe:v1
and in grafana:
Frontend and Backend
Now the next step in th exercise:
to send requests to user-->frontend--> backend; we'll use the /hostz endpoint to do that. Remember, the /hostz endpoint takes a frontend request, sends it to the backend which inturn echos back the podName the backend runs as. The entire response is then returned to the user. This is just a way to show the which backend host processed the requests.
(note i'm using jq utility to parse JSON)
for i in {1..1000}; do curl -s -k https://$GATEWAY_IP/hostz | jq '.[0].body'; sleep 1; doneyou should see output indicating traffic from the v1 backend verison: be-v1-*. Thats what we expect since our original rule sets defines only fe:v1 and be:v1 as valid targets.
$ for i in {1..1000}; do curl -s -k https://$GATEWAY_IP/hostz | jq '.[0].body'; sleep 1; done
"pod: [be-v1-58855cf854-t6rz9] node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v1-58855cf854-t6rz9] node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v1-58855cf854-t6rz9] node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v1-58855cf854-t6rz9] node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v1-58855cf854-t6rz9] node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v1-58855cf854-t6rz9] node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"Note both Kiali and Grafana shows both frontend and backend service telemetry and traffic to be:v1
Route Control
This section details how to selectively send traffic to specific service versions and control traffic routing.
Selective Traffic
In this sequence, we will setup a routecontrol to:
- Send all traffic to
myapp:v1. - traffic from
myapp:v1can only go tobe:v2
Basically, this is a convoluted way to send traffic from fe:v1-> be:v2 even if all services and versions are running.
The yaml on istio-fev1-bev2.yaml would direct inbound traffic for myapp:v1 to go to be:v2 based on the sourceLabels:. The snippet for this config is:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: be-virtualservice
spec:
gateways:
- mesh
hosts:
- be
http:
- match:
- sourceLabels:
app: myapp
version: v1
route:
- destination:
host: be
subset: v2
weight: 100
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: be-destination
spec:
host: be
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
loadBalancer:
simple: ROUND_ROBIN
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2So lets apply the config with kubectl:
kubectl replace -f istio-fev1-bev2.yamlAfter sending traffic, check which backend system was called by invoking /hostz endpoint on the frontend.
What the /hostz endpoint does is takes a users request to fe-* and targets any be-* that is valid. Since we only have fe-v1 instances running and the fact we setup a rule such that only traffic from fe:v1 can go to be:v2, all the traffic outbound for be-* must terminate at a be-v2:
$ for i in {1..1000}; do curl -s -k https://$GATEWAY_IP/hostz | jq '.[0].body'; sleep 1; done
"pod: [be-v2-8475cfb896-wxt4z] node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v2-8475cfb896-wxt4z] node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v2-8475cfb896-wxt4z] node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v2-8475cfb896-wxt4z] node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v2-8475cfb896-wxt4z] node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v2-8475cfb896-wxt4z] node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v2-8475cfb896-wxt4z] node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"and on the frontend version is always one.
for i in {1..100}; do curl -k https://$GATEWAY_IP/version; sleep 1; done
11111111111111111111111111111Note the traffic to be-v1 is 0 while there is a non-zero traffic to be-v2 from fe-v1:
Look at the incoming requests by source graph:
If we now overlay rules that direct traffic allow interleaved fe(v1|v2) -> be(v1|v2) we expect to see requests to both frontend v1 and backend
kubectl replace -f istio-fev1v2-bev1v2.yaml
then frontend is both v1 and v2:
for i in {1..1000}; do curl -k https://$GATEWAY_IP/version; sleep 1; done
111211112122211121212211122211and backend is responses comes from both be-v1 and be-v2
$ for i in {1..1000}; do curl -s -k https://$GATEWAY_IP/hostz | jq '.[0].body'; sleep 1; done
"pod: [be-v1-58855cf854-t6rz9] node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z] node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v1-58855cf854-t6rz9] node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z] node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v1-58855cf854-t6rz9] node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z] node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v1-58855cf854-t6rz9] node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
Route Path
Now lets setup a more selective route based on a specific path in the URI:
- The rule we're defining is: "First route requests to myapp where
path=/versionto only go to thev1set"...if there is no match, fall back to the default routes where you send20%traffic tov1and80%traffic tov2
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: myapp-virtualservice
spec:
hosts:
- "*"
gateways:
- my-gateway
http:
- match:
- uri:
exact: /version
route:
- destination:
host: myapp
subset: v1
- route:
- destination:
host: myapp
subset: v1
weight: 20
- destination:
host: myapp
subset: v2
weight: 80kubectl replace -f istio-route-version-fev1-bev1v2.yamlSo check all requests to /version are fe:v1
for i in {1..1000}; do curl -k https://$GATEWAY_IP/version; sleep 1; done
1111111111111111111
You may have noted how the route to any other endpoint other than /version destination is weighted split and not delcared round robin (eg:)
- route:
- destination:
host: myapp
subset: v1
weight: 20
- destination:
host: myapp
subset: v2
weight: 80Anyway, now lets edit rule to and change the prefix match to /xversion so the match doesn't apply. What we expect is a request to http://gateway_ip/version will go to v1 and v2 (since the path rule did not match and the split is the fallback rule.
kubectl replace -f istio-route-version-fev1-bev1v2.yaml
Observe the version of the frontend you're hitting:
for i in {1..1000}; do curl -k https://$GATEWAY_IP/version; sleep 1; done
2121212222222222222221122212211222222222222222What you're seeing is myapp-v1 now getting about 20% of the traffic while myapp-v2 gets 80% because the previous rule doens't match.
Undo that change /xversion --> /version and reapply to baseline:
kubectl replace -f istio-route-version-fev1-bev1v2.yaml
Canary Deployments with VirtualService
You can use this traffic distribuion mechanism to run canary deployments between released versions. For example, a rule like the following will split the traffic between v1|v2 at 80/20 which you can use to gradually roll traffic over to v2 by applying new percentage weights.
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: myapp-virtualservice
spec:
hosts:
- "*"
gateways:
- my-gateway
- my-gateway-ilb
http:
- route:
- destination:
host: myapp
subset: v1
weight: 80
- destination:
host: myapp
subset: v2
weight: 20Destination Rules
Lets configure Destination rules such that all traffic from myapp-v1 round-robins to both version of the backend.
First lets force all gateway requests to go to v1 only:
on istio-fev1-bev1v2.yaml:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: myapp-virtualservice
spec:
hosts:
- "*"
gateways:
- my-gateway
- my-gateway-ilb
http:
- route:
- destination:
host: myapp
subset: v1And where the backend trffic is split between be-v1 and be-v2 with a ROUND_ROBIN
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: be-destination
spec:
host: be
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
loadBalancer:
simple: ROUND_ROBIN
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2After you apply the rule,
kubectl replace -f istio-fev1-bev1v2.yaml
you'll see frontend request all going to fe-v1
for i in {1..1000}; do curl -k https://$GATEWAY_IP/version; sleep 1; done
11111111111111with backend requests coming from pretty much round robin
$ for i in {1..1000}; do curl -s -k https://$GATEWAY_IP/hostz | jq '.[0].body'; sleep 1; done
"pod: [be-v1-58855cf854-t6rz9] node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z] node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v1-58855cf854-t6rz9] node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z] node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v1-58855cf854-t6rz9] node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z] node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v1-58855cf854-t6rz9] node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z] node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v1-58855cf854-t6rz9] node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z] node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v2-8475cfb896-wxt4z] node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"Now change the istio-fev1-bev1v2.yaml to RANDOM and see response is from v1 and v2 random:
$ for i in {1..1000}; do curl -s -k https://$GATEWAY_IP/hostz | jq '.[0].body'; sleep 1; done
"pod: [be-v1-58855cf854-t6rz9] node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z] node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v1-58855cf854-t6rz9] node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v1-58855cf854-t6rz9] node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z] node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v2-8475cfb896-wxt4z] node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v1-58855cf854-t6rz9] node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z] node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v1-58855cf854-t6rz9] node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v1-58855cf854-t6rz9] node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z] node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v1-58855cf854-t6rz9] node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z] node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"
"pod: [be-v1-58855cf854-t6rz9] node: [gke-cluster-1-default-pool-7907ec5d-jrwq]"
"pod: [be-v2-8475cfb896-wxt4z] node: [gke-cluster-1-default-pool-7907ec5d-nxvx]"Internal LoadBalancer
The configuration here sets up an internal loadbalancer on GCP to access an exposed istio service.
The config settings that enabled this during istio setup was done by an operator and annotation:
Specifically, we created a new ingressGateway and set its annotation to
cloud.google.com/load-balancer-type: "internal"
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
components:
ingressGateways:
- name: istio-ingressgateway
enabled: true
k8s:
service:
ports:
- name: https
port: 443
protocol: TCP
- name: istio-ilbgateway
enabled: true
k8s:
serviceAnnotations:
cloud.google.com/load-balancer-type: "internal"
service:
ports:
- port: 443
name: https
protocol: TCPWe did that duirng setup and later on, attached a Gateway to it which also exposed only :443
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: my-gateway-ilb
spec:
selector:
istio: istio-ilbgateway
servers:
- port:
number: 443
name: https
protocol: HTTPS
hosts:
- "*"
tls:
mode: SIMPLE
serverCertificate: /etc/istio/ilbgateway-certs/tls.crt
privateKey: /etc/istio/ilbgateway-certs/tls.key We also specified a VirtualService which selected these inbound gateways to the myapp service: This configuration was defined when we applied istio-fev1-bev1.yaml:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: myapp-virtualservice
spec:
hosts:
- "*"
gateways:
- my-gateway
- my-gateway-ilb
http:
- route:
- destination:
host: myapp
subset: v1
weight: 100Note the gateways: entry in the VirtualService includes my-gateway-ilb which is what defines host:myapp, subset:v1 as a target for the ILB
gateways:
- my-gateway
- my-gateway-ilbAs mentioned above, we had to manually specify the port the ILB will listen on for traffic inbound to this service. \
Finally, the certficates Secret mounted at /etc/istio/ilbgateway-certs/ was specified this in the initial all-istio.yaml file:
apiVersion: v1
data:
tls.crt: LS0tLS1CR...
tls.key: LS0tLS1CR...
kind: Secret
metadata:
name: istio-ilbgateway-certs
namespace: istio-system
type: kubernetes.io/tlsNow that the service is setup, acquire the ILB IP allocated
export ILB_GATEWAY_IP=$(kubectl -n istio-system get service istio-ilbgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo $ILB_GATEWAY_IP
Then from a GCE VM in the same VPC, send some traffic over on the internal address
$ curl -vk https://10.128.0.122/
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=California; O=Google; OU=Enterprise; CN=gke.default.svc.cluster.local
* start date: Dec 24 18:17:46 2017 GMT
* expire date: Jun 11 18:17:46 2020 GMT
* issuer: C=US; ST=California; L=Mountain View; O=Google; OU=Enterprise; CN=TestCAforESO
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET / HTTP/2
> Host: 10.128.0.122
> User-Agent: curl/7.64.0
> Accept: */*
< HTTP/2 200
< x-powered-by: Express
< content-type: text/html; charset=utf-8
< content-length: 19
< etag: W/"13-tsbq4e7agwVV6r9iE+Lb/lLwlzw"
< date: Tue, 22 Sep 2020 16:24:55 GMT
< x-envoy-upstream-service-time: 3
< server: istio-envoy
Hello from Express!- The Kiali console should show traffic from both gateways (if you recently sent traffic in externally and internally):
Egress Rules
By default, istio blocks the cluster from making outbound requests. There are several options to allow your service to connect externally:
- Egress Rules
- Egress Gateway
- Setting
global.proxy.includeIPRanges
Egress rules prevent outbound calls from the server except with whiteliste addresses.
For example:
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: bbc-ext
spec:
hosts:
- www.bbc.com
ports:
- number: 80
name: http
protocol: HTTP
resolution: DNS
location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: google-ext
spec:
hosts:
- www.google.com
ports:
- number: 443
name: https
protocol: HTTPS
resolution: DNS
location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: google-ext
spec:
hosts:
- www.google.com
tls:
- match:
- port: 443
sni_hosts:
- www.google.com
route:
- destination:
host: www.google.com
port:
number: 443
weight: 100
Allows only http://www.bbc.com/* and https://www.google.com/*
To test the default policies, the /requestz endpoint tries to fetch the following URLs:
var urls = [
'https://www.google.com/robots.txt',
'http://www.bbc.com/robots.txt',
'http://www.google.com:443/robots.txt',
'https://www.cornell.edu/robots.txt',
'https://www.uwo.ca/robots.txt',
'http://www.yahoo.com/robots.txt'
]First make sure there is an inbound rule already running:
kubectl replace -f istio-fev1-bev1.yamlAnd that you're using REGISTRY_ONLY:
kubectl get configmap istio -n istio-system -o yaml | grep -o "mode: REGISTRY_ONLY"- Without egress rule, requests will fail:
curl -k -s https://$GATEWAY_IP/requestz | jq '.'gives
[
{
"url": "https://www.google.com/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: Client network socket disconnected before secure TLS connection was established",
},
{
"url": "http://www.google.com:443/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
},
{
"url": "http://www.bbc.com/robots.txt",
"body": "",
"statusCode": 502
},
{
"url": "https://www.cornell.edu/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
},
{
"url": "https://www.uwo.ca/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: Client network socket disconnected before secure TLS connection was established",
},
{
"url": "https://www.yahoo.com/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: Client network socket disconnected before secure TLS connection was established",
},
{
"url": "http://www.yahoo.com:443/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
]
Note: the
502response for thebbc.comentry is the actual denial rule from the istio-proxy (502->Bad Gateway)
then apply the egress policy which allows www.bbc.com:80 and www.google.com:443
kubectl apply -f istio-egress-rule.yaml
gives
curl -s -k https://$GATEWAY_IP/requestz | jq '.'[
{
"url": "https://www.google.com/robots.txt",
"statusCode": 200
},
{
"url": "http://www.google.com:443/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
},
{
"url": "http://www.bbc.com/robots.txt",
"statusCode": 200
},
{
"url": "https://www.cornell.edu/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
},
{
"url": "https://www.uwo.ca/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
},
{
"url": "https://www.yahoo.com/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
},
{
"url": "http://www.yahoo.com:443/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
]
Notice that only one of the hosts worked over SSL worked
Egress Gateway
THe egress rule above initiates the proxied connection from each sidecar....but why not initiate the SSL connection from a set of bastion/egress
gateways we already setup? THis is where the Egress Gateway configurations come up but inorder to use this: The following configuration will allow egress traffic for www.yahoo.com via the gateway. See HTTPS Egress Gateway
So.. lets revert the config we setup above
kubectl delete -f istio-egress-rule.yaml
then lets apply the rule for the gateway:
kubectl apply -f istio-egress-gateway.yamlNotice the gateway TLS mode is PASSTHROUGH ("Note the PASSTHROUGH TLS mode which instructs the gateway to pass the ingress traffic AS IS, without terminating TLS.")
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: istio-egressgateway
spec:
selector:
istio: egressgateway
servers:
- port:
number: 443
name: tls
protocol: TLS
hosts:
- www.yahoo.com
tls:
mode: PASSTHROUGHcurl -s -k https://$GATEWAY_IP/requestz | jq '.'[
{
"url": "https://www.google.com/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
},
{
"url": "http://www.google.com:443/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
},
{
"url": "http://www.bbc.com/robots.txt",
"body": "",
"statusCode": 502
},
{
"url": "https://www.cornell.edu/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
},
{
"url": "https://www.uwo.ca/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
},
{
"url": "https://www.yahoo.com/robots.txt",
"statusCode": 200 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
},
{
"url": "http://www.yahoo.com:443/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
}
}
]
You can also tail the egress gateway logs:
$ kubectl logs -f --tail=0 -l istio=egressgateway -n istio-system
[2020-04-29T15:23:39.949Z] "- - -" 0 - "-" "-" 829 5706 144 - "-" "-" "-" "-" "72.30.35.10:443" outbound|443||www.yahoo.com 10.12.1.4:57332 10.12.1.4:443 10.12.2.10:41592 www.yahoo.com -
[2020-04-29T15:23:48.195Z] "- - -" 0 - "-" "-" 829 5722 138 - "-" "-" "-" "-" "98.138.219.231:443" outbound|443||www.yahoo.com 10.12.1.4:40632 10.12.1.4:443 10.12.2.10:41658 www.yahoo.com -TLS Origination for Egress Traffic
In this mode, traffic exits the pod unencrypted but gets proxied via the gateway for an https destination. For this to work, traffic must originate from the pod unencrypted but specify the port as an SSL port. In current case, if you want to send traffic for https://www.yahoo.com/robots.txt, emit the request from the pod as http://www.yahoo.com:443/robots.txt. Note the traffic is http:// and the port is specified: :443
Ok, lets try it out, apply:
kubectl apply -f istio-egress-gateway-tls-origin.yamlThen notice just the last, unencrypted traffic to yahoo succeeds
curl -s -k https://$GATEWAY_IP/requestz | jq '.'
[
{
"url": "https://www.google.com/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: Client network socket disconnected before secure TLS connection was established",
}
},
{
"url": "http://www.google.com:443/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
}
},
{
"url": "http://www.bbc.com/robots.txt",
"body": "",
"statusCode": 502
},
{
"url": "https://www.cornell.edu/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: Client network socket disconnected before secure TLS connection was established",
},
{
"url": "https://www.uwo.ca/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: Client network socket disconnected before secure TLS connection was established",
}
},
{
"url": "http://www.yahoo.com/robots.txt",
"statusCode": 200
},
{
"url": "https://www.yahoo.com/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: Client network socket disconnected before secure TLS connection was established",
}
},
{
"url": "http://www.yahoo.com:443/robots.txt",
"statusCode": {
"name": "RequestError",
"message": "Error: read ECONNRESET",
}
}
]
Bypass Envoy entirely
You can also configure the global.proxy.includeIPRanges= variable to completely bypass the IP ranges for certain serivces. This setting is described under Calling external services directly and details the ranges that should get covered by the proxy. For GKE, you need to cover the subnets included and allocated:
Access GCE MetadataServer
The /metadata endpoint access the GCE metadata server and returns the current projectID. This endpoint makes three separate requests using the three formats I've see GCP client libraries use. (note: the hostnames are supposed to resolve to the link local IP address shown below)
app.get('/metadata', (request, response) => {
var resp_promises = []
var urls = [
'http://metadata.google.internal/computeMetadata/v1/project/project-id',
'http://metadata/computeMetadata/v1/project/project-id',
'http://169.254.169.254/computeMetadata/v1/project/project-id'
]So if you make an inital request, you'll see 404 errors from Envoy since we did not setup any rules.
[
{
"url": "http://metadata.google.internal/computeMetadata/v1/project/project-id",
"body": "",
"statusCode": 502
},
{
"url": "http://metadata/computeMetadata/v1/project/project-id",
"body": "",
"statusCode": 502
},
{
"url": "http://169.254.169.254/computeMetadata/v1/project/project-id",
"body": "",
"statusCode": 502
}
]So lets do just that:
kubectl apply -f istio-egress-rule-metadata.yaml
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: metadata-ext
spec:
addresses:
- 169.254.169.254
hosts:
- metadata.google.internal
ports:
- number: 80
name: http
protocol: HTTP
resolution: STATIC
location: MESH_EXTERNAL
endpoints:
- address: 169.254.169.254Try it again and you should see
[
{
"url": "http://metadata.google.internal/computeMetadata/v1/project/project-id",
"statusCode": 200
},
{
"url": "http://metadata/computeMetadata/v1/project/project-id",
"statusCode": 502
},
{
"url": "http://169.254.169.254/computeMetadata/v1/project/project-id",
"statusCode": 200
}
]
WebAssembly
Ref: Redefining extensibility in proxies - introducing WebAssembly to Envoy and Istio
The following steps will deploy a trivial wasm module to the cluster that returns hello world back as a header.
We are going to compile and use the main upstream istio envoy example for wasm. The expected output is a new response header and replaced location attribute
see envoy_filter_http_wasm_example.cc
To use this, you need bazel installed and ready to compile a c++ app from scratch
git clone --single-branch --branch release-1.7 https://github.com/istio/envoy.git
bazel build examples/wasm:envoy_filter_http_wasm_example.wasmOnce you have that, upload the binary as a config map called example-filter
kubectl create cm -n default example-filter --from-file=`pwd`/bazel-bin/examples/wasm/envoy_filter_http_wasm_example.wasm
Then direct inte sidecar to use a mounted wasm filter
kubectl apply -f fe-v1-wasm-inject.yaml
kubectl replace -f fe-v1-wasm.yaml
Once thats done, invoke the frontend:
$ curl -vk https://$GATEWAY_IP/version
> GET /version HTTP/2
> Host: 34.123.13.130
> user-agent: curl/7.72.0
> accept: */*
< HTTP/2 200
< x-powered-by: Express
< content-type: text/html; charset=utf-8
< content-length: 1
< etag: W/"1-NWoZK3kTsExUV00Ywo1G5jlUKKs"
< date: Tue, 22 Sep 2020 18:08:35 GMT
< x-envoy-upstream-service-time: 38
< newheader: newheadervalue
< location: envoy-wasm <<<<<<<<<<<<<<<<<
< server: istio-envoy <<<<<<<<<<<<<<<<<
< You'll see the two new headers.
TODO: use wasme cli
LUA HTTPFilter
The following will setup a simple Request/Response LUA EnvoyFilter for the frontent myapp:
The settings below injects headers in both the request and response streams:
kubectl apply -f istio-fev1-httpfilter-lua.yaml
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: ui-examplefilter
namespace: default
spec:
configPatches:
- applyTo: HTTP_FILTER
match:
context: SIDECAR_INBOUND
proxy:
proxyVersion: '1\.7.*'
listener:
filterChain:
filter:
name: envoy.http_connection_manager
subFilter:
name: envoy.router
patch:
operation: INSERT_BEFORE
value:
name: envoy.filters.http.lua
typed_config:
'@type': type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
inlineCode: |
function envoy_on_request(request_handle)
request_handle:headers():add("foo", "bar")
end
function envoy_on_response(response_handle)
response_handle:headers():add("foo2", "bar2")
end
workloadSelector:
labels:
app: myapp
version: v1Note the response headers back to the caller (foo2:bar2) and the echo of the headers as received by the service from envoy (foo:bar)
$ curl -vk https://$GATEWAY_IP/headerz
> GET /headerz HTTP/2
> Host: 35.184.101.110
> User-Agent: curl/7.60.0
> Accept: */*
< HTTP/2 200
< x-powered-by: Express
< content-type: application/json; charset=utf-8
< contLUAent-length: 626
< etag: W/"272-vkps3sJOT8NW67CxK6gzGw"
< date: Fri, 22 Mar 2019 00:40:36 GMT
< x-envoy-upstream-service-time: 7
< foo2: bar2 <<<<<<<<<
< server: istio-envoy
{
"host": "34.123.13.130",
"user-agent": "curl/7.72.0",
"accept": "*/*",
"x-forwarded-for": "10.128.0.121",
"x-forwarded-proto": "https",
"x-request-id": "52fb996a-2f7a-9352-be5e-025c4328bcae",
"x-envoy-attempt-count": "1",
"content-length": "0",
"x-envoy-internal": "true",
"x-forwarded-client-cert": "By=spiffe://cluster.local/ns/default/sa/myapp-sa;Hash=3cb34c33d650cdc2b9c62ad7d617be39dd778ec34895e03a1a405137a0fcf3f0;Subject=\"\";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account",
"foo": "bar", <<<<<<<<<<<<<<<<
"x-b3-traceid": "ec14fdbbe9ffd9d0499bc785667ed21f",
"x-b3-spanid": "63d7f435af04f296",
"x-b3-parentspanid": "499bc785667ed21f",
"x-b3-sampled": "1"
}You can also see the backend request header by running an echo back of those headers
curl -v -k https://$GATEWAY_IP/hostz
< HTTP/2 200
< x-powered-by: Express
< content-type: application/json; charset=utf-8
< content-length: 168
< etag: W/"a8-+rQK5xf1qR07k9sBV9qawQ"
< date: Fri, 22 Mar 2019 00:44:30 GMT
< x-envoy-upstream-service-time: 33
< foo2: bar2 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
< server: istio-envoy
Authorization and Authorization
The following steps is basically another walkthrough of the RequestAuthentication and AuthorizationPolicy
JWT Authentication and RBAC Authorization
In this section, we will extend the sample to implement JWT authentication from the client and also use claims within the JWT payload for an enhanced Service Specific Policy.
Specifically, this section will add perimeter Authentication that validates a JWT token at ingress gateway and then RBAC policies at the Service level will further restrict requests.
There are two users: Alice, Bob and two services svc1, svc2. Alice should be allowed to access only svc1, Bob should only access svc2. Both users must present a JWT issued by the same issuer. In this case, a Self Signed JWT certificate issued by Google. You can also use Fireabase/Cloud Identity or any other JWT that provides a JWK URL)
This section involves several steps...first delete any configurations that may still be active. We need to do this because we will create two new services on the frontend svc1, svc2
kubectl delete -f istio-fev1-httpfilter-lua.yaml
kubectl delete -f istio-fev1-httpfilter-ext_authz.yaml
kubectl delete -f istio-fev1-bev1v2.yaml
kubectl delete -f all-istio.yaml
kubectl apply -f istio-lb-certs.yaml
kubectl apply -f istio-ingress-gateway.yaml
kubectl apply -f istio-ingress-ilbgateway.yaml You can verify the configuration that are active by running:
$ kubectl get svc,deployments,po,serviceaccounts,serviceentry,VirtualService,DestinationRule,Secret,GatewaySince the authentication mode described here involes a JWT, we will setup a Google Cloud Service Account the JWT provider. You are ofcourse free to use any identity provide or even Firebase/Cloud Identity
First redeploy an application that has two frontend services svc1, svc2 accessible using the Host: headervalues (svc1.domain.com and svc2.domain.com)
cd auth_rbac_policy/
kubectl apply -f auth-deployment.yaml -f istio-fe-svc1-fe-svc2.yaml
Check the application still works (it should; we didn't apply policies yet yet)
curl -k -H "Host: svc1.example.com" -w "\n" https://$GATEWAY_IP/versionApply the authentication policy that checks for a JWT signed by the service account and audience match on the service. THe following policy will allow all three audience values through the ingress gateway but only those JWTs that match the audience for the service through at the service level:
To bootstrap all this, first we need some JWTS. In this case, we will use GCP serice accounts
To bootstrap the sample client, go to the Google Cloud Console and download a service account JSON file as described [here](https://cloud.google.com/iam/docs/creating-managing-service-account-keys). Copy the service account to the `auth_rbac_policy/jwt_cli` folder and save the JSON file as `svc_account.json`.
First get the name of the serice account that will sign the JWT:
```bash
cd auth_rbac_policy/jwt_cli/
export PROJECT_ID=`gcloud config get-value core/project`
gcloud iam service-accounts create sa-istio --display-name "JWT issuer for Istio helloworld"
export SA_EMAIL=sa-istio@$PROJECT_ID.iam.gserviceaccount.com
gcloud iam service-accounts keys create svc_account.json --iam-account=$SA_EMAIL
echo SA_EMAIL
Edit auth-policy.yaml file and replace the values where the service account email $SA_EMAIL is specified
After you apply the policy
kubectl apply -f auth-policy.yaml
Note that by default we have mTLS and deny by default
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: deny-all-authz-ns
spec:
{}
---
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: default-peerauth
namespace: default
spec:
mtls:
mode: STRICTmake an api call with a malformed authentication header:
$ curl -k -H "Host: svc1.example.com" -H "Authorization: Bearer foo" -w "\n" https://$GATEWAY_IP/version
Jwt is not in the form of Header.Payload.Signaturenow try without a header entirely:
$ curl -k -H "Host: svc1.example.com" -w "\n" https://$GATEWAY_IP/version
RBAC: access deniedThe error indicates we did not send in the required header. In the next setp, we will use a small sample client library to acquire a JWT. You can also use google OIDC tokens or any other provider (Firebase, Auth0)
The policy above looks for a specific issuer and audience value. THe jwksUri field maps to the public certificate set for our service account. THe well known url for the service account for Google Cloud is:
https://www.googleapis.com/service_accounts/v1/jwk/<serviceAccountEmail>
pip install -r requirements.txt
python main.pyThe command line utility will generate two tokens with different specifications. For Alice, .
{
"iss": "source-service-account@fabled-ray-104117.iam.gserviceaccount.com",
"iat": 1571185182,
"sub": "alice",
"exp": 1571188782,
"aud": "https://svc1.example.com"
}And Bob
{
"groups": [
"group1",
"group2"
],
"sub": "bob",
"exp": 1571188782,
"iss": "source-service-account@fabled-ray-104117.iam.gserviceaccount.com",
"iat": 1571185182,
"aud": "https://svc2.example.com"
}Bob, no groups
{
"iss": "source-service-account@fabled-ray-104117.iam.gserviceaccount.com",
"iat": 1571185734,
"sub": "bob",
"exp": 1571189334,
"aud": "https://svc2.example.com"
}Export these values as environment variables
export TOKEN_ALICE=<tokenvaluealice>
export TOKEN_BOB=<tokenvaluebob>
export TOKEN_BOB_NO_GROUPS=<tokenvaluebob2>
WARNING the sample code to generate the jwt at the client side uses a service account JWT where the client itself is minting the JWT specifications (meaning it can setup any claimsets it wants, any
subfield.). In reality, you wouild want to use some other mechanism to acquire a token (Auth0, Firebase Custom Claims, etc).
Now inject the token into the Authorization: Bearer header and try to access the protected service:
for i in {1..1000}; do curl -k -H "Host: svc1.example.com" -H "Authorization: Bearer $TOKEN_ALICE" -w "\n" https://$GATEWAY_IP/version; sleep 1; doneThe request should now pass validation and you're in. What we just did is have one policy that globally to the ingress-gateway. Note, we also applied per-service policies in auth-policy.yaml that checks for the aud: value in the inbound token.
What that means is if you use Alice's token to access svc2, you'll see an authentication validation error because that token doesn't have "https://svc2.example.com" in the audience
$ curl -k -H "Host: svc2.example.com" -H "Authorization: Bearer $TOKEN_ALICE" -w "\n" https://$GATEWAY_IP/version
Audiences in Jwt are not allowed
In our example, we had a self-signed JWT locally meaning if the end-user had a service account capable of singing, they coudl setup any audience value (i.,e Alice could create a JWT token with the audience of svc). We need to back up and apply addtional controls through RBAC.
Authorization using JWT Claims
The other way is to push the allow/deny decision down from Authentication to Authorization and then using claims on the Authz polic
In auth-policy.yaml, uncomment the AuthorizationPolicy stanza which does the first check for correct audience value in the inbound token for svc2 (the one bob uses):
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: svc2-az
spec:
action: ALLOW
selector:
matchLabels:
app: svc2
rules:
- to:
- operation:
methods: ["GET"]
when:
- key: request.auth.claims[iss]
values: ["sa-istio@mineral-minutia-820.iam.gserviceaccount.com"]
- key: request.auth.claims[aud]
values: ["https://svc2.example.com"]
# - key: request.auth.claims[groups]
# values: ["group1", "group2"]
- key: request.auth.claims[sub]
values: ["bob"]
kubectl apply -f auth-policy.yaml
Consider we have two JWT tokens for `Bob`:
One with groups
```json
{
"groups": [
"group1",
"group2"
],
"sub": "bob",
"exp": 1571188782,
"iss": "source-service-account@fabled-ray-104117.iam.gserviceaccount.com",
"iat": 1571185182,
"aud": "https://svc2.example.com"
}
And one without
{
"iss": "source-service-account@fabled-ray-104117.iam.gserviceaccount.com",
"iat": 1571185734,
"sub": "bob",
"exp": 1571189334,
"aud": "https://svc2.example.com"
}Both Tokens allow access through to the serivce because they pass authentication (the audience and subject):
$ curl -sk -H "Host: svc2.example.com" -H "Authorization: Bearer $TOKEN_BOB" -o /dev/null -w "%{http_code}\n" https://$GATEWAY_IP/version
200
$ curl -sk -H "Host: svc2.example.com" -H "Authorization: Bearer $TOKEN_BOB_NO_GROUPS" -o /dev/null -w "%{http_code}\n" https://$GATEWAY_IP/version
200But what we want to do is deny a request if the token does not include the group header (i know, if Bob had the service account file, he could "just set it"...anyway)
For now, edit auth-policy.yaml and modify the authorization policy for the backend service to make sure the groups are specified and the groups claims are set
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: svc2-az
spec:
selector:
matchLabels:
app: svc2
rules:
- to:
- operation:
methods: ["GET"]
when:
- key: request.auth.claims[iss]
values: ["sa-istio@mineral-minutia-820.iam.gserviceaccount.com"]
- key: request.auth.claims[aud]
values: ["https://svc2.example.com"]
- key: request.auth.claims[groups]
values: ["group1", "group2"]
- key: request.auth.claims[sub]
values: ["bob"]Apply again,
kubectl apply -f auth-policy.yaml
Wait maybe 30seconds (it takes time for the policy to propagte)
Once you set that, only Alice should be able to access svc1 and only Bob access svc2 except when no group info is provided in the JWT
$ curl -sk -H "Host: svc1.example.com" -H "Authorization: Bearer $TOKEN_ALICE" -w "%{http_code}\n" https://$GATEWAY_IP/version
200
$ curl -sk -H "Host: svc2.example.com" -H "Authorization: Bearer $TOKEN_ALICE" -w "%{http_code}\n" https://$GATEWAY_IP/version
403
Audiences in Jwt are not allowed
$ curl -sk -H "Host: svc1.example.com" -H "Authorization: Bearer $TOKEN_BOB" -w "%{http_code}\n" https://$GATEWAY_IP/version
403
Audiences in Jwt are not allowed
$ curl -sk -H "Host: svc2.example.com" -H "Authorization: Bearer $TOKEN_BOB" -o /dev/null -w "%{http_code}\n" https://$GATEWAY_IP/version
200
$ curl -sk -H "Host: svc2.example.com" -H "Authorization: Bearer $TOKEN_BOB_NO_GROUPS" -o /dev/null --w "%{http_code}\n" https://$GATEWAY_IP/version
403
RBAC: access deniedNotice that bob was only allowed in when the token carried group info.
Service to Service and Authentication Policy
In this section, we extend the working set to allow Alice and Bob to access frontend services and ALSO setup an RBAC policy that allows svcA to access svcB.
When we deployed the application, we associated a service account with each workoad
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: svc1-sa
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: svc2-sa
---We can use this service acount to say: 'only allow requests from svc1-sa to access svc2'. We do this by placing another AuthorizationPolicy policy rule in for svc2
- from:
- source:
principals: ["cluster.local/ns/default/sa/svc1-sa"] That is,
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: svc2-az
spec:
selector:
matchLabels:
app: svc2
rules:
- from:
- source:
principals: ["cluster.local/ns/default/sa/svc1-sa"]
to:
- operation:
methods: ["GET"]
when:
- key: request.auth.claims[iss]
values: ["sa-istio@mineral-minutia-820.iam.gserviceaccount.com"]
- key: request.auth.claims[aud]
values: ["https://svc2.example.com"]
- key: request.auth.claims[groups]
values: ["group1", "group2"]
- key: request.auth.claims[sub]
values: ["bob"]Apply again,
kubectl apply -f auth-policy.yaml
Now, if bob tries to access svc2 externally even with a correct token, he will see
$ curl -sk -H "Host: svc2.example.com" -H "Authorization: Bearer $TOKEN_BOB" -w "%{http_code}\n" https://$GATEWAY_IP/version
RBAC: access deniedLet try to exc into a pod where svc1 is running and access svc2:
$ kubectl get po
NAME READY STATUS RESTARTS AGE
svc1-77b6bd69cc-bldf8 2/2 Running 0 29m
svc2-bcf6cbd55-hrp25 2/2 Running 0 29m
$ kubectl exec -ti svc1-77b6bd69cc-bldf8 -- /bin/bashFirst try to access the backend service:
curl -s -w "%{http_code}\n" http://svc2.default.svc.cluster.local:8080/version
403
RBAC: access deniedYou'll see a 403 because although the request was inbound from svc1 which is using PEER authentication, we did not add Bob's JWT token. So set an env-var and execute the request again:
root@svc1-7489fbf8d4-8tffm:/# export TOKEN_BOB=eyJhbGciOi...
root@svc1-7489fbf8d4-8tffm:/# curl -s -w "%{http_code}\n" -H "Authorization: Bearer $TOKEN_BOB" http://svc2.default.svc.cluster.local:8080/version
200
You are now in!
This is a bit silly since we needed to use the JWT token for bob for just service to serice traffic.
You dont' ofcourse need to do that: just edit the AuthorizationPolicy for svc2 and comment out
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: svc2-az
spec:
action: ALLOW
selector:
matchLabels:
app: svc2
rules:
- to:
- operation:
methods: ["GET"]
from:
- source:
principals: ["cluster.local/ns/default/sa/svc1-sa"]
# when:
# - key: request.auth.claims[iss]
# values: ["sa-istio@mineral-minutia-820.iam.gserviceaccount.com"]
# - key: request.auth.claims[aud]
# values: ["https://svc2.example.com"]
# - key: request.auth.claims[groups]
# values: ["group1", "group2"]
# - key: request.auth.claims[sub]
# values: ["bob"]Bob can't access svc2 from the outside but svc1 can access svc2
# apply
$ kubectl apply -f auth-policy.yaml
# from external -> svc1
curl -sk -H "Host: svc2.example.com" -H "Authorization: Bearer $TOKEN_BOB" -w "%{http_code}\n" https://$GATEWAY_IP/version
403
RBAC: access denied
# from svc1->svc
curl -s -w "%{http_code}\n" http://svc2.default.svc.cluster.local:8080/version
200External Authorization HTTPFilter
You can also setup envoy.ext_authz Filter in this cluster. When using the ext_authz filter on the frontend service, any request for app: myapp, version: v1 will undergo an external authorization check by a serivce you run elsewhere. The external serivice will only allow a request through if it carries Authorizaton: Bearer foo in the header.
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: ext-authz-filter
spec:
workloadSelector:
labels:
app: myapp
version: v1
configPatches:
- applyTo: HTTP_FILTER
match:
proxy:
proxyVersion: ^1\.7.*
context: SIDECAR_INBOUND
listener:
portNumber: 8080
filterChain:
filter:
name: "envoy.http_connection_manager"
subFilter:
name: "envoy.router"
patch:
operation: INSERT_BEFORE
value:
name: envoy.filters.http.ext_authz
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
grpc_service:
google_grpc:
target_uri: "your_grpc_server_ip:50051"
stat_prefix: "ext_authz" To use this type of authorization check, you will need to run a serivce somewhere (either within istio (strongly preferred for latency) or external to istio). The following runs the serivce external to istio:
- Istio External Authorization Server
- Envoy External Authorization server (envoy.ext_authz) HelloWorld
First spin up a GCP VM that has an external IP, install golang there and startup the authz server in the git repo provided. You'll also need to open up port :50051 to that VM.
After that, add in the ip address of yoru vm to the yaml file and apply the envoy filter:
kubectl apply -f istio-fev1-httpfilter-ext_authz.yamlOnce you do that, every request to the fronend service will fail unless the specific header is sent through.
Note you'll ofcourse not want to run this serivce anywhere thas externally accessible!...this is just for a demo!!
THis is what an inbound request from istio to the authorization server may look like:
$ go run grpc_server.go
2020/02/20 20:41:40 Starting gRPC Server at :50051
2020/02/20 20:42:41 >>> Authorization called check()
2020/02/20 20:42:41 Inbound Headers:
2020/02/20 20:42:41 {
":authority": "35.238.81.95",
":method": "GET",
":path": "/version",
"accept": "*/*",
"authorization": "Bearer foo",
"content-length": "0",
"user-agent": "curl/7.66.0",
"x-b3-sampled": "0",
"x-b3-spanid": "b228f9e2179794c5",
"x-b3-traceid": "725c448565d59423b228f9e2179794c5",
"x-envoy-internal": "true",
"x-forwarded-client-cert": "By=spiffe://cluster.local/ns/default/sa/myapp-sa;Hash=ae6b57b6ce2932c74b54c40c5e1a7a13daf2828edeb688f9d273a6ea54f38dbf;Subject=\"\";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account",
"x-forwarded-for": "10.128.0.61",
"x-forwarded-proto": "https",
"x-istio-attributes": "CiMKGGRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZRIHEgVteWFwcAoqCh1kZXN0aW5hdGlvbi5zZXJ2aWNlLm5hbWVzcGFjZRIJEgdkZWZhdWx0Ck4KCnNvdXJjZS51aWQSQBI+a3ViZXJuZXRlczovL2lzdGlvLWluZ3Jlc3NnYXRld2F5LTk3ZGNkN2Y4Ny02MnpjZC5pc3Rpby1zeXN0ZW0KPQoYZGVzdGluYXRpb24uc2VydmljZS5ob3N0EiESH215YXBwLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwKOwoXZGVzdGluYXRpb24uc2VydmljZS51aWQSIBIeaXN0aW86Ly9kZWZhdWx0L3NlcnZpY2VzL215YXBw",
"x-request-id": "e7932678-b3a2-40b8-bc49-6b645448ae28"
}Cleanup
The easiest way to clean up what you did here is to delete the GKE cluster!
gcloud container clusters delete cluster-1
Conclusion
The steps i outlined above is just a small set of what Istio has in store. I'll keep updating this as it move towards 1.0 and subsequent releases.