Redirect loop using redirect-from-regex annotation
Nikarous opened this issue · comments
Description of the problem
I am setting up redirects for 3 domains with redirect-from-regex annotations on k8s ingress object:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
...
cert-manager.io/cluster-issuer: letsencrypt
acme.cert-manager.io/http01-edit-in-place: "true"
ingress.kubernetes.io/redirect-from-code: '301'
ingress.kubernetes.io/redirect-from-regex: >-
^www\.service\.dev\.example\.com$|^service-test\.example\.com$|^www\.service-test\.example\.com$
kubernetes.io/ingress.class: haproxy
spec:
rules:
- host: service.dev.example.com
http:
paths:
- backend:
service:
name: example-service
port:
number: 80
path: /
pathType: Prefix
tls:
- hosts:
- service.dev.example.com
- service-test.example.com
- www.service-test.example.com
- www.service.dev.example.com
secretName: lets-cert
And reloading configuration (in case of scaling for example) setups redirects randomly to one of those 4 domains with message:
W0612 09:12:56.334065 7 host.go:107] ignoring regex redirect from '^www\.service\.dev\.example\.com$|^service-test\.example\.com$|^www\.service-test\.example\.com$' on Ingress 'example-ns/ingress-controller-for-example-service', it's already targeting to 'www.service-test.example.com'
I0612 09:13:47.721941 7 instance.go:441] updating 4 host(s): [service.dev.example.com service-test.example.com www.service-test.example.com www.service.dev.example.com]
And its not refreshing setup to point redirects to service.dev.example.com. This message repeats when new pod is added and after some time it manages to lands correctly on hostname setup in ingress.
Important notes:
I did some testing and at first I thought that | (or) symbol in regex were causing issues but it still persist in second example of ingress below.
Now I am pretty sure that for some reason HAproxy-ingress takes host values for redirect map from hosts array under tls. This is because it puts values in the _front_redir_from__regex.map that it could not know other than from this field in ingress resource, example in second ingress below.
example I got directly from my haproxy-ingress pod:
/etc/haproxy # cat maps/_front_redir_from__regex.map
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# #
# # HAProxy Ingress Controller
# # --------------------------
# # This file is automatically updated, do not edit
# #
#
^w{0,3}\.{0,1}service-test1\.example\.com$ marketplace.test1.example.com
^w{0,3}\.{0,1}service-test2\.example\.com$ www.service-test2.example.com
^w{0,3}\.{0,1}service-test3\.example\.com$ marketplace.test3.example.com
Expected behavior
HAproxy creates _front_redir_from__regex.map with correct values of regex and hostname from ingress rules host field
Steps to reproduce the problem
- Create ingress object similar to the one above
- Scale service enforcing configuration reload
- Repeat few times, configuration in _front_redir_from__regex.map will show different hostnames sometimes.
Environment information
HAProxy Ingress version: v0.14.3
Command-line options:
--configmap=ingress/haproxy-ingress
--ingress-class=haproxy
--sort-backends
Global options:
Name: haproxy-ingress
Namespace: ingress
Labels: app.kubernetes.io/instance=haproxy-ingress
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=haproxy-ingress
app.kubernetes.io/version=v0.14.3
helm.sh/chart=haproxy-ingress-0.14.3
Annotations: meta.helm.sh/release-name: haproxy-ingress
meta.helm.sh/release-namespace: ingress
Data
====
healthz-port:
----
10253
prometheus-port:
----
9101
stats-port:
----
1936
BinaryData
====
Events: <none>
Ingress objects:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
...
cert-manager.io/cluster-issuer: letsencrypt
acme.cert-manager.io/http01-edit-in-place: "true"
ingress.kubernetes.io/redirect-from-code: '301'
ingress.kubernetes.io/redirect-from-regex: >-
^www\.service\.dev\.example\.com$|^service-test\.example\.com$|^www\.service-test\.example\.com$
kubernetes.io/ingress.class: haproxy
spec:
rules:
- host: service.dev.example.com
http:
paths:
- backend:
service:
name: example-service
port:
number: 80
path: /
pathType: Prefix
tls:
- hosts:
- service.dev.example.com
- service-test.example.com
- www.service-test.example.com
- www.service.dev.example.com
secretName: lets-cert
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
...
cert-manager.io/cluster-issuer: letsencrypt
acme.cert-manager.io/http01-edit-in-place: "true"
ingress.kubernetes.io/redirect-from-code: '301'
ingress.kubernetes.io/redirect-from-regex: >-
^w{0,3}\.{0,1}service-test\.example\.com$
kubernetes.io/ingress.class: haproxy
spec:
rules:
- host: service.dev.example.com
http:
paths:
- backend:
service:
name: example-service
port:
number: 80
path: /
pathType: Prefix
tls:
- hosts:
- service.dev.example.com
- service-test.example.com
- www.service-test.example.com
- www.service.dev.example.com
secretName: lets-cert
Hello, I fixed the issue with this configuration:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
ingress.kubernetes.io/redirect-from-code: '301'
ingress.kubernetes.io/redirect-from-regex: >-
^www\.service\.dev\.example\.com$|^service-test\.example\.com$|^www\.service-test\.example\.com$
kubernetes.io/ingress.class: haproxy
spec:
rules:
- host: service.dev.example.com
http:
paths:
- backend:
service:
name: example-service
port:
number: 80
path: /
pathType: Prefix
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
cert-manager.io/cluster-issuer: letsencrypt
acme.cert-manager.io/http01-edit-in-place: "true"
kubernetes.io/ingress.class: haproxy
spec:
rules:
- host: service.dev.example.com
http:
paths:
- backend:
service:
name: example-service
port:
number: 80
path: /
pathType: Prefix
tls:
- hosts:
- service.dev.example.com
- service-test.example.com
- www.service-test.example.com
- www.service.dev.example.com
secretName: lets-cert
as you can see the issue was caused because code adds hosts from TLS just as the host from rules so it also takes redirect annotations which is not what I needed.
Hi, sorry the delay, finally took the time to have a proper look on that issue. You're right, the problem is that hosts configured via TLS are in fact being added as regular hosts. We need this behavior in place in order to make haproxy use the right cert when handling the TLS handshake. All the downstream host processing is harmless, except redirect, which needs a controlled ingress manifest configuration.
Just added #1010 that should fix it. We'll have new tags later this week.
Regarding the fix you provided, I'm surprised that it's working. redirect-from config is host scoped, so the annotation will be fully applied on that host despite it's being configured on a distinct manifest. As another option you could try reverting the logic and using redirect-to instead, which shouldn't have this problem.
Fixed on v0.13.13 and v0.14.4. Closing.