Goo Edge fails to process incoming requests after ratelimit/extauth upstream settings misconfiguration

[Dmitriy Zakomirnyi]

Gloo Edge Product

Open Source

Gloo Edge Version

v1.16.9

Kubernetes Version

v1.27.4

Describe the bug

A gateway fails to process any incoming requests after ratelimit/extauth upstream misconfiguration in a Settings resource.
All requests are affected despite their corresponding route ratelimit/authorization configuration options.

Expected Behavior

The gateway should not fail request processing, if no ratelimit/extauth policies applied.

Steps to reproduce the bug

1. Install Gloo Edge gateway
2. Install Pets application

kubectl apply -f https://raw.githubusercontent.com/solo-io/gloo/v1.16.x/example/petstore/petstore.yaml

3. Setup the application route

kubectl apply -f https://gist.githubusercontent.com/dmi3zkm/f85a850bfebb1e63ced67c1c1a177c03/raw/d1ecaea8bf14c1e0fbb4810f1facb516260d77a2/vs.yaml

4. Verify the route

curl -i 'http://proxy-server-url.com/all-pets' -H 'Host: rl.solo.io'

HTTP/1.1 200 OK
content-type: application/xml
date: Fri, 29 Mar 2024 16:35:14 GMT
content-length: 86
x-envoy-upstream-service-time: 7
server: envoy

[{"id":1,"name":"Dog","status":"available"},{"id":2,"name":"Cat","status":"pending"}]

5. Install the rate limit service

kubectl apply -f https://gist.githubusercontent.com/dmi3zkm/862d7438e3f634f8546d371ada643ded/raw/b76214b98bb2dde7e424bd44b13b626e8288265a/rl-setup.yaml

6. Verify the rate limit service installation
7. Configure the rate limit server in Gloo Edge Settings resource.

apiVersion: gloo.solo.io/v1
kind: Settings
metadata:
  ...
spec:
  ...
  ratelimitServer:
    rateLimitBeforeAuth: false
    ratelimitServerRef:
      name: rl-ratelimit-8081
      namespace: gloo-system

8. Curl the route. The result should be 200 OK
9. Point the gateway to the non-existing rate limit server upstream

apiVersion: gloo.solo.io/v1
kind: Settings
metadata:
  ...
spec:
  ...
  ratelimitServer:
    rateLimitBeforeAuth: false
    ratelimitServerRef:
      name: rrl-ratelimit-8081
      namespace: gloo-system

10. Curl your route again.

curl -i 'http://proxy-server-url.com/all-pets' -H 'Host: rl.solo.io'
curl: (52) Empty reply from server

Additional Environment Detail

No response

Additional Context

No response

[Dmitriy Zakomirnyi]

Update:

After investigation I've found out the reason is the snippet of code below, which is located at https://github.com/solo-io/gloo/blob/main/projects/gloo/pkg/plugins/ratelimit/plugin.go#L119

// projects/gloo/pkg/plugins/ratelimit/plugin.go

func (p *plugin) HttpFilters(params plugins.Params, listener *v1.HttpListener) ([]plugins.StagedHttpFilter, error) {
	serverSettings := p.getServerSettingsForListener(listener)

	upstreamRef := serverSettings.GetRatelimitServerRef()
	if upstreamRef == nil {
		return nil, nil
	}

	// Make sure the server exists
	_, err := params.Snapshot.Upstreams.Find(upstreamRef.GetNamespace(), upstreamRef.GetName())
	if err != nil {
		return nil, ServerNotFound(upstreamRef)
	}

The returned err is appended to the HTTP Listener error report. That's the reason why validator fails the translation.
And that leads me to a question is it always strictly necessary to fail the whole listener if there is no valid rate limit upstream?
Is it a good idea to make this behavior configurable?