kubeseal is ignoring NO_PROXY variables upon request for cert.pem (403)
kriipke opened this issue · comments
Which component:
version 0.25
of sealed secrets, tried with both 0.25
and 0.26
of the kubeseal
client
Describe the bug
kubeseal is ignoring NO_PROXY variables upon request to:
https://10.20.157.251:6443/api/v1/namespaces/sealed-secrets/services/http:eksa-dev-sealed-secrets:http/proxy/v1/cert.pem
Other workloads on the cluster are using the no_proxy vars just fine so we know they're set correctly on the nodes. Any thoughts?
To Reproduce
Steps to reproduce the behavior:
cat /tmp/flame.yaml | kubeseal --controller-name eksa-dev-sealed-secrets --controller-namespace sealed-secrets -v9
I0307 16:48:30.122752 9784 loader.go:395] Config loaded from file: /home/spencer/.kube/config.new
I0307 16:48:30.124765 9784 round_trippers.go:466] curl -v -XGET -H "Accept: application/x-pem-file, */*" -H "User-Agent: kubeseal/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://10.20.157.251:6443/api/v1/namespaces/sealed-secrets/services/eksa-dev-sealed-secrets'
I0307 16:48:30.138387 9784 round_trippers.go:510] HTTP Trace: Dial to tcp:10.20.157.251:6443 succeed
I0307 16:48:30.181576 9784 round_trippers.go:553] GET https://10.20.157.251:6443/api/v1/namespaces/sealed-secrets/services/eksa-dev-sealed-secrets 200 OK in 56 milliseconds
I0307 16:48:30.181642 9784 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 13 ms TLSHandshake 15 ms ServerProcessing 27 ms Duration 56 ms
I0307 16:48:30.181648 9784 round_trippers.go:577] Response Headers:
I0307 16:48:30.181679 9784 round_trippers.go:580] Cache-Control: no-cache, private
I0307 16:48:30.181713 9784 round_trippers.go:580] Content-Type: application/json
I0307 16:48:30.181717 9784 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: e1f30f59-a886-4c1e-87d7-0fe2b1babbe2
I0307 16:48:30.181732 9784 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: c4fea2d7-140b-4b07-819f-f43ed6866ef4
I0307 16:48:30.181747 9784 round_trippers.go:580] Content-Length: 2445
I0307 16:48:30.181750 9784 round_trippers.go:580] Date: Thu, 07 Mar 2024 21:48:29 GMT
I0307 16:48:30.181752 9784 round_trippers.go:580] Audit-Id: 5a2cebf6-d50f-4e48-b356-86189188b0bf
I0307 16:48:30.181834 9784 request.go:1212] Response Body: {"kind":"Service","apiVersion":"v1","metadata":{"name":"eksa-dev-sealed-secrets","namespace":"sealed-secrets","uid":"17616c06-db33-4efc-9b9d-827465e8c4ee","resourceVersion":"2770070","creationTimestamp":"2024-03-07T20:14:19Z","labels":{"app.kubernetes.io/instance":"release-name","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"sealed-secrets","app.kubernetes.io/part-of":"sealed-secrets","app.kubernetes.io/version":"0.25.0","argocd.argoproj.io/instance":"sealedsecrets-devdw01","helm.sh/chart":"sealed-secrets-2.14.2"},"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"app.kubernetes.io/instance\":\"release-name\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"sealed-secrets\",\"app.kubernetes.io/part-of\":\"sealed-secrets\",\"app.kubernetes.io/version\":\"0.25.0\",\"argocd.argoproj.io/instance\":\"sealedsecrets-devdw01\",\"helm.sh/chart\":\"sealed-secrets-2.14.2\"},\"name\":\"eksa-dev-sealed-secrets\",\"namespace\":\"sealed-secrets\"},\"spec\":{\"ports\":[{\"name\":\"http\",\"nodePort\":null,\"port\":8080,\"targetPort\":\"http\"}],\"selector\":{\"app.kubernetes.io/instance\":\"release-name\",\"app.kubernetes.io/name\":\"sealed-secrets\"},\"type\":\"ClusterIP\"}}\n"},"managedFields":[{"manager":"argocd-controller","operation":"Update","apiVersion":"v1","time":"2024-03-07T20:14:19Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}},"f:labels":{".":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/name":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:argocd.argoproj.io/instance":{},"f:helm.sh/chart":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":8080,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}}}]},"spec":{"ports":[{"name":"http","protocol":"TCP","port":8080,"targetPort":"http"}],"selector":{"app.kubernetes.io/instance":"release-name","app.kubernetes.io/name":"sealed-secrets"},"clusterIP":"10.180.172.15","clusterIPs":["10.180.172.15"],"type":"ClusterIP","sessionAffinity":"None","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","internalTrafficPolicy":"Cluster"},"status":{"loadBalancer":{}}}
I0307 16:48:30.182524 9784 round_trippers.go:466] curl -v -XGET -H "Accept: application/x-pem-file, */*" -H "User-Agent: kubeseal/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://10.20.157.251:6443/api/v1/namespaces/sealed-secrets/services/http:eksa-dev-sealed-secrets:http/proxy/v1/cert.pem'
I0307 16:48:30.198596 9784 round_trippers.go:553] GET https://10.20.157.251:6443/api/v1/namespaces/sealed-secrets/services/http:eksa-dev-sealed-secrets:http/proxy/v1/cert.pem 403 Forbidden in 15 milliseconds
I0307 16:48:30.198667 9784 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 15 ms Duration 15 ms
I0307 16:48:30.198672 9784 round_trippers.go:577] Response Headers:
I0307 16:48:30.198696 9784 round_trippers.go:580] Vary: Accept-Language
I0307 16:48:30.198700 9784 round_trippers.go:580] Via: 1.1 devqa-proxy01 (squid/4.10)
I0307 16:48:30.198702 9784 round_trippers.go:580] X-Cache: MISS from devqa-proxy01
I0307 16:48:30.198704 9784 round_trippers.go:580] Audit-Id: db7f2df2-55b8-4c59-b771-974118a5a706
I0307 16:48:30.198706 9784 round_trippers.go:580] Content-Language: en
I0307 16:48:30.198708 9784 round_trippers.go:580] Content-Type: text/html;charset=utf-8
I0307 16:48:30.198711 9784 round_trippers.go:580] Mime-Version: 1.0
I0307 16:48:30.198726 9784 round_trippers.go:580] Server: squid/4.10
I0307 16:48:30.198728 9784 round_trippers.go:580] X-Squid-Error: ERR_ACCESS_DENIED 0
I0307 16:48:30.198732 9784 round_trippers.go:580] Cache-Control: no-cache, private
I0307 16:48:30.198734 9784 round_trippers.go:580] Date: Thu, 07 Mar 2024 21:48:29 GMT
I0307 16:48:30.198735 9784 round_trippers.go:580] X-Cache-Lookup: NONE from devqa-proxy01:3128
I0307 16:48:30.198737 9784 round_trippers.go:580] Content-Length: 3902
You'll notice the X-Squid-Error: ERR_ACCESS_DENIED 0
in the response headers on the request to https://10.20.157.251:6443/api/v1/namespaces/sealed-secrets/services/http:eksa-dev-sealed-secrets:http/proxy/v1/cert.pem
.
Below are the settings we have set for no_proxy
and NO_PROXY
we have set on every Kubernetes node:
Expected behavior
A clear and concise description of what you expected to happen.
cat /tmp/secret.yaml | kubeseal --controller-name eksa-dev-sealed-secrets --controller-namespace sealed-secrets -v9
I0307 16:39:45.619156 6205 loader.go:395] Config loaded from file: /home/spencer/.kube/config.new
I0307 16:39:45.621467 6205 round_trippers.go:466] curl -v -XGET -H "Accept: application/x-pem-file, */*" -H "User-Agent: kubeseal/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://10.20.156.251:6443/api/v1/namespaces/sealed-secrets/services/eksa-dev-sealed-secrets'
I0307 16:39:45.636424 6205 round_trippers.go:510] HTTP Trace: Dial to tcp:10.20.156.251:6443 succeed
I0307 16:39:45.702002 6205 round_trippers.go:553] GET https://10.20.156.251:6443/api/v1/namespaces/sealed-secrets/services/eksa-dev-sealed-secrets 200 OK in 80 milliseconds
I0307 16:39:45.702038 6205 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 14 ms TLSHandshake 51 ms ServerProcessing 13 ms Duration 80 ms
I0307 16:39:45.702042 6205 round_trippers.go:577] Response Headers:
I0307 16:39:45.702047 6205 round_trippers.go:580] Cache-Control: no-cache, private
I0307 16:39:45.702049 6205 round_trippers.go:580] Content-Type: application/json
I0307 16:39:45.702051 6205 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: 026c84a0-5672-439c-bd1b-d124a0c84a2b
I0307 16:39:45.702053 6205 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: 946d256b-75a1-4a03-ade4-d33c50fdb1ff
I0307 16:39:45.702055 6205 round_trippers.go:580] Content-Length: 2443
I0307 16:39:45.702056 6205 round_trippers.go:580] Date: Thu, 07 Mar 2024 21:39:41 GMT
I0307 16:39:45.702058 6205 round_trippers.go:580] Audit-Id: 3c66040d-9424-4fa5-a34a-0e186a82e9f8
I0307 16:39:45.702091 6205 request.go:1212] Response Body: {"kind":"Service","apiVersion":"v1","metadata":{"name":"eksa-dev-sealed-secrets","namespace":"sealed-secrets","uid":"84672da6-aff6-4912-a117-2712d93c37cd","resourceVersion":"2028736","creationTimestamp":"2024-02-28T19:40:10Z","labels":{"app.kubernetes.io/instance":"release-name","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"sealed-secrets","app.kubernetes.io/part-of":"sealed-secrets","app.kubernetes.io/version":"0.25.0","argocd.argoproj.io/instance":"sealedsecrets-devpw01","helm.sh/chart":"sealed-secrets-2.14.2"},"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"app.kubernetes.io/instance\":\"release-name\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"sealed-secrets\",\"app.kubernetes.io/part-of\":\"sealed-secrets\",\"app.kubernetes.io/version\":\"0.25.0\",\"argocd.argoproj.io/instance\":\"sealedsecrets-devpw01\",\"helm.sh/chart\":\"sealed-secrets-2.14.2\"},\"name\":\"eksa-dev-sealed-secrets\",\"namespace\":\"sealed-secrets\"},\"spec\":{\"ports\":[{\"name\":\"http\",\"nodePort\":null,\"port\":8080,\"targetPort\":\"http\"}],\"selector\":{\"app.kubernetes.io/instance\":\"release-name\",\"app.kubernetes.io/name\":\"sealed-secrets\"},\"type\":\"ClusterIP\"}}\n"},"managedFields":[{"manager":"argocd-controller","operation":"Update","apiVersion":"v1","time":"2024-02-28T19:40:10Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}},"f:labels":{".":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/name":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:argocd.argoproj.io/instance":{},"f:helm.sh/chart":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":8080,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}}}]},"spec":{"ports":[{"name":"http","protocol":"TCP","port":8080,"targetPort":"http"}],"selector":{"app.kubernetes.io/instance":"release-name","app.kubernetes.io/name":"sealed-secrets"},"clusterIP":"10.180.49.50","clusterIPs":["10.180.49.50"],"type":"ClusterIP","sessionAffinity":"None","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","internalTrafficPolicy":"Cluster"},"status":{"loadBalancer":{}}}
I0307 16:39:45.702484 6205 round_trippers.go:466] curl -v -XGET -H "Accept: application/x-pem-file, */*" -H "User-Agent: kubeseal/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://10.20.156.251:6443/api/v1/namespaces/sealed-secrets/services/http:eksa-dev-sealed-secrets:http/proxy/v1/cert.pem'
I0307 16:39:45.720084 6205 round_trippers.go:553] GET https://10.20.156.251:6443/api/v1/namespaces/sealed-secrets/services/http:eksa-dev-sealed-secrets:http/proxy/v1/cert.pem 200 OK in 17 milliseconds
I0307 16:39:45.720125 6205 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 17 ms Duration 17 ms
I0307 16:39:45.720129 6205 round_trippers.go:577] Response Headers:
I0307 16:39:45.720144 6205 round_trippers.go:580] Date: Thu, 07 Mar 2024 21:39:35 GMT
I0307 16:39:45.720157 6205 round_trippers.go:580] Content-Length: 1724
I0307 16:39:45.720159 6205 round_trippers.go:580] Audit-Id: 4c45100f-aaf0-4b4c-9123-6f58342eacdf
I0307 16:39:45.720172 6205 round_trippers.go:580] Cache-Control: no-cache, private
I0307 16:39:45.720184 6205 round_trippers.go:580] Content-Type: application/x-pem-file
{
"kind": "SealedSecret",
"apiVersion": "bitnami.com/v1alpha1",
"metadata": {
"name": "flame-vars",
"namespace": "default",
"creationTimestamp": null
},
"spec": {
"template": {
"metadata": {
"name": "flame-vars",
"namespace": "default",
"creationTimestamp": null
}
},
"encryptedData": {
"PASSWORD": "AgAX0yRADabrY1lb51sWFXgVpeMeSrvsmY/BukWOeHdyNZ+f+FB6goUNDO59WariGxLkWr7s7tnrInjsnvQ2/cpVkHGdgcVNgwZSiuz+dyOm2cK6RTeF5/2T/O+2XMKQMIx5JzFk9dnsBGuOrLAlWwJ6F0qBkgCkLH/iGdXbUYlpUst7UjUi2PXlI4hOtQVpipOE3/Dr+WFES8/b0Pqu9lScniBOajaebWQRdw99wACK+CPALPtCNP7XF6ffXU4/uCilOu732GeZERjHB8JFQXcoFt6zJhz+9LKmt9qfXZCLi2lQLVAnZ/Qr+rYjDTaly9uLWMqXrXODoxherWWGZWDxhYEQ2ayfblbe+0XK9bKa+ei57jUkNRlYUqpQs2lEuWHeYx+HlVPRpijOthSQF8ZcYwrCGz7n3Vf+PEdIbFJdD06sK8nYa/zKZd4nUbVKzehHMcdgsTyR5HbYON9JjiRA63t6lea5ploDKswo9RxQpFX3/chyfggJp0guAbpUuT9XHinnK8kdpbFANmn5UThiDuyuM3TuQDd7GqBlHuzUVGsrVCsGy5axwIkdn3mlCOVNcvi0kOMgXX1E6JZx3azUgA7fawihKAWPXuhAPIGHJduKdaK2OQ0CkJlKEGZ+Eg4KuRZ1VNI3KPC3+CfGaXT97KyDzozBNIq82EEL3KdgWBgoTqeRpMJ9i3dNg8aesRXMnL4tWxOMOA=="
}
}
}
Version of Kubernetes:
- Output of
kubectl version
:
Client Version: v1.27.6-eks-aab2b08
Kustomize Version: v5.0.1
Server Version: v1.27.4-eks-cedffd4
Additional context
Do you have the NO_PROXY
variable set on your local environment? I assume you are using kubeseal
from your local environment and not from inside the cluster, so kubeseal
will get the proxy settings from your local environment.
It is set, however, the machine I'm on shouldn't be using the proxy at all..
The reason we shared the cluster info is that the same kubeseal
client will work on one of our clusters but not on the one I'm posting about, so we figured it must be a difference in how the clusters are configured. However, when we look at the sealed-secrets-controller logs it looks like the kubeseal
requests aren't even making it to the cluster because they're going through the proxy.
Super weird?
It is indeed weird, because I see two requests GET requests in your logs but the first one is succeeding, so it seems the NO_PROXY
variable is functioning correctly there.
Can you try with an older version so we can see if there's something different? For example, 0.24.5.
yeah! it is weird, we saw the exact same thing, we're not sure why the second request is all of a sudden ignoring the NO_PROXY var.
We will give 0.24.5
a try, thanks!