Fail to setup authservice at Ingress Gateway level for Istio 1.10.3
anhdle14 opened this issue · comments
I am struggling to setup auth-service at Ingress Gateway level.
Environments
- istio v1.10.3
- authservice v0.4.1
- kubectl 1.21.3 | server 1.21.2
Applied Manifest
# IstioOperator
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
name: istio
namespace: istio-system
spec:
components:
ingressGateways:
- enabled: true
k8s:
overlays:
- kind: Deployment
name: istio-ingressgateway
patches:
- path: spec.template.spec.containers
value:
- args:
- proxy
- router
- --domain
- $(POD_NAMESPACE).svc.cluster.local
- --proxyLogLevel=warning
- --proxyComponentLogLevel=misc:error
- --log_output_level=default:info
- --serviceCluster
- istio-ingressgateway
env:
- name: JWT_POLICY
value: third-party-jwt
- name: PILOT_CERT_PROVIDER
value: istiod
- name: CA_ADDR
value: istiod.istio-system.svc:15012
- name: NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: INSTANCE_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: HOST_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
- name: SERVICE_ACCOUNT
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.serviceAccountName
- name: CANONICAL_SERVICE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.labels['service.istio.io/canonical-name']
- name: CANONICAL_REVISION
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.labels['service.istio.io/canonical-revision']
- name: ISTIO_META_WORKLOAD_NAME
value: istio-ingressgateway
- name: ISTIO_META_OWNER
value: kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-ingressgateway
- name: ISTIO_META_MESH_ID
value: cluster.local
- name: TRUST_DOMAIN
value: cluster.local
- name: ISTIO_META_UNPRIVILEGED_POD
value: "true"
- name: ISTIO_META_ROUTER_MODE
value: standard
- name: ISTIO_META_CLUSTER_ID
value: Kubernetes
image: docker.io/istio/proxyv2:1.10.3
imagePullPolicy: IfNotPresent
name: istio-proxy
ports:
- containerPort: 15021
protocol: TCP
- containerPort: 8080
protocol: TCP
- containerPort: 8443
protocol: TCP
- containerPort: 31400
protocol: TCP
- containerPort: 15443
protocol: TCP
- containerPort: 15090
name: http-envoy-prom
protocol: TCP
readinessProbe:
failureThreshold: 30
httpGet:
path: /healthz/ready
port: 15021
scheme: HTTP
initialDelaySeconds: 1
periodSeconds: 2
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: "2"
memory: 1Gi
requests:
cpu: 10m
memory: 40Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
privileged: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/istio/proxy
name: istio-envoy
- mountPath: /etc/istio/config
name: config-volume
- mountPath: /var/run/secrets/istio
name: istiod-ca-cert
- mountPath: /var/run/secrets/tokens
name: istio-token
readOnly: true
- mountPath: /var/lib/istio/data
name: istio-data
- mountPath: /etc/istio/pod
name: podinfo
- mountPath: /etc/istio/ingressgateway-certs
name: ingressgateway-certs
readOnly: true
- mountPath: /etc/istio/ingressgateway-ca-certs
name: ingressgateway-ca-certs
readOnly: true
- name: authservice
image: ghcr.io/istio-ecosystem/authservice/authservice:0.4.1
imagePullPolicy: Always
ports:
- containerPort: 10003
volumeMounts:
- name: auth-service-cm
mountPath: /etc/authservice
- path: spec.template.spec.volumes
value:
- configMap:
name: auth-service-cm
name: auth-service-cm
- configMap:
defaultMode: 420
name: istio-ca-root-cert
name: istiod-ca-cert
- downwardAPI:
defaultMode: 420
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.labels
path: labels
- fieldRef:
apiVersion: v1
fieldPath: metadata.annotations
path: annotations
- path: cpu-limit
resourceFieldRef:
containerName: istio-proxy
divisor: 1m
resource: limits.cpu
- path: cpu-request
resourceFieldRef:
containerName: istio-proxy
divisor: 1m
resource: requests.cpu
name: podinfo
- emptyDir: {}
name: istio-envoy
- emptyDir: {}
name: istio-data
- name: istio-token
projected:
defaultMode: 420
sources:
- serviceAccountToken:
audience: istio-ca
expirationSeconds: 43200
path: istio-token
- configMap:
defaultMode: 420
name: istio
optional: true
name: config-volume
- name: ingressgateway-certs
secret:
defaultMode: 420
optional: true
secretName: istio-ingressgateway-certs
- name: ingressgateway-ca-certs
secret:
defaultMode: 420
optional: true
secretName: istio-ingressgateway-ca-certs
service:
ports:
- name: status-port
port: 15021
protocol: TCP
targetPort: 15021
- name: http2
port: 80
protocol: TCP
targetPort: 8080
- name: https
port: 443
protocol: TCP
targetPort: 8443
- name: tcp
port: 31400
protocol: TCP
targetPort: 31400
- name: tls
port: 15443
protocol: TCP
targetPort: 15443
name: istio-ingressgateway
label:
version: v1.10.3
profile: demo
tag: 1.10.3# auth-service.yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: auth-service-cm
namespace: istio-system
data:
config.json: |
{
"listen_address": "127.0.0.1",
"listen_port": "10003",
"log_level": "trace",
"threads": 8,
"chains": [
{
"name": "idp_filter_chain",
"filters": [
{
"oidc": {
"authorization_uri": "https://a.auth0.com/authorize",
"token_uri": "https://a.auth0.com/oauth/token",
"callback_uri": "https://*.anhdle14.me/oauth/callback",
"jwks": "---REDACTED BUT THE SAME AS https://YOUR_DOMAIN/.well-known/jwks.json ---",
"client_id": "xxx",
"client_secret": "xxx",
"scopes": [],
"cookie_name_prefix": "istio",
"id_token": {
"preamble": "Bearer",
"header": "Authorization"
},
"logout": {
"path": "/authservice_logout",
"redirect_uri": "https://*.anhdle14.me/logout/"
}
}
}
]
}
]
}
---
# Patch for 1.10.3 and the deprecation of V2 xDS.
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: ingress-token-service-filter
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: HTTP_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: envoy.filters.network.http_connection_manager
subFilter:
name: "envoy.filters.http.jwt_authn"
patch:
operation: INSERT_BEFORE
value:
name: envoy.filters.http.ext_authz
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
stat_prefix: ext_authz
transport_api_version: V3
grpc_service:
envoy_grpc:
cluster_name: ext_authz
timeout: 10s # Timeout for the entire request (including authcode for token exchange with the IDP)
- applyTo: CLUSTER
match:
context: ANY
cluster: {} # this line is required starting in istio 1.4.0
patch:
operation: ADD
value:
name: ext_authz
connect_timeout: 5s # This timeout controls the initial TCP handshake timeout - not the timeout for the entire request
type: LOGICAL_DNS
lb_policy: ROUND_ROBIN
http2_protocol_options: {}
load_assignment:
cluster_name: ext_authz
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 10003
---
# JWT AuthN rule
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
name: ingressgateway
namespace: istio-system
spec:
selector:
matchLabels:
istio: ingressgateway
jwtRules:
- issuer: https://anhdle14.jp.auth0.com/
jwksUri: https://anhdle14.jp.auth0.com/.well-known/jwks.json
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: ingressgateway
namespace: istio-system
spec:
selector:
matchLabels:
istio: ingressgateway
action: ALLOW
rules:
- from
- source:
requestPrincipals: ["*"]
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: gateway
namespace: istio-system
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*.anhdle14.me"
tls:
httpsRedirect: true
- port:
number: 443
name: https
protocol: HTTPS
hosts:
- "*.anhdle14.me"
tls:
mode: SIMPLE
credentialName: wildcard-ssl-secretThere are some notes that I do not understand:
The path part of the configured
callback_uri, and if configured, thelogout.pathmust be paths that are routable to the app by the ingress gateway's VirtualService, or else the Authservice at the gateway will not receive those requests and will not be able to process them. This is perhaps counter-intuitive since the Authservice container is no longer running in the app's pod, so it may not feel like this is needed, but it is required.
I am testing this with httpbin. But from my understanding this is enough already.
# httpbin.vs.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: httpbin
spec:
hosts:
- "httpbin.anhdle14.me"
gateways:
- istio-system/gateway
http:
- route:
- destination:
port:
number: 8000
host: httpbinResult / Error
But the result is just RBAC: access denied for all the VS under ingress gateway.
I have tried to tinker with AuthorizationPolicy, but I couldn't fix anything. I can tell you all the scenarios I have tried but the overall result is it doesn't work at all.
I have tested the sidecar deployment and it was working fine. So I don't know what went south with ingressgateway deployment.
thanks. this will be captured in #149
The PR to fix 1.10 is merged now. Closing this. Let me know if you see this error again.