Default chart values break outbound HTTPS in Istio clusters
jhuffman-wyn opened this issue · comments
Discovered a weird one today. If you have the following setup:
- Keda installed with admission webhook (default settings)
- Istio installed with applications using Istio sidecar containers
...all outbound HTTPS traffic will be blocked with mysterious "SSL protocol failure" and similar errors. This is due to an unpleasant interaction between keda and Istio, and while the onus may be on Istio to fix, keda can also proactively avoid contributing to it.
Expected Behavior
HTTPS traffic is able to reach external endpoints without issue.
Actual Behavior
All HTTPS traffic on pods with Istio sidecars is blocked with TLS/SSL protocol errors.
Steps to Reproduce the Problem
- Install Istio (any recent version) with default settings.
- Install current keda version with default settings.
- Have at least one pod with an Istio sidecar.
- From within that pod, attempt to reach any external HTTPS endpoint.
- You will get a TLS/SSL error.
Note that this blog is what informed me of the issue: https://fable.sh/blog/istio-port-443-and-ssl-errors./
That's not specific to keda, but it is apparent that the Istio fix did not account for every scenario: istio/istio#16458
This is being logged as a keda bug as changing the admission webhook service can easily avoid causing this problem:
apiVersion: v1
kind: Service
metadata:
name: keda-admission-webhooks
spec:
...
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: http
port: 443
protocol: TCP
targetPort: 9443
selector:
app: keda-admission-webhooks
sessionAffinity: None
type: ClusterIP
The specific combination of the port name being http and the port being 443 causes this. Changing either of those values will avoid this. Calling it http-web, for instance, will do the job just fine.
Specifications
- KEDA Version: 2.11.1
- Platform & Version:
- Kubernetes Version: GKE 1.24
- Scaler(s): N/A
Interesting...
Are you willing to contribute with the fix?
nvm
@congzhegao you need to open a PR from your fork, so fork this repo, commit the change there to some branch and then open PR from the branch on your fork.
btw, this should be also fixed on the core https://github.com/kedacore/keda/blob/main/config/webhooks/service.yaml
Both PRs are merged, so I close the issue as solved. Changes will be released soon (we plan to cut a release these weeks)