DoS on guard-service

[Roland Huß]

with a simple loop that curls with random URL, guard-service fails to update the Guardian CR because of size limitations:

Script:

/ # while true
> do
> i=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 50 | head -n 1)
> curl http://helloworld-go.default/$i?$i
> done

results after ~ 10-20 seconds in the following logoutput:

knative-serving guard-service-6cb5b4bc6-g5h8d guard-service info	Failed to update KubeApi with new config default.helloworld-go: set crd ns default sid helloworld-go: updating resource Request entity too large: limit is 3145728
knative-serving guard-service-6cb5b4bc6-g5h8d guard-service info	Failed to update KubeApi with new config default.helloworld-go: set crd ns default sid helloworld-go: updating resource Request entity too large: limit is 3145728
knative-serving guard-service-6cb5b4bc6-g5h8d guard-service info	Failed to update KubeApi with new config default.helloworld-go: set crd ns default sid helloworld-go: updating resource Request entity too large: limit is 3145728
knative-serving guard-service-6cb5b4bc6-g5h8d guard-service info	Failed to update KubeApi with new config default.helloworld-go: set crd ns default sid helloworld-go: updating resource Request entity too large: limit is 3145728
knative-serving guard-service-6cb5b4bc6-g5h8d guard-service info	Failed to update KubeApi with new config default.helloworld-go: set crd ns default sid helloworld-go: updating resource Request entity too large: limit is 3145728
knative-serving guard-service-6cb5b4bc6-g5h8d guard-service info	Failed to update KubeApi with new config default.helloworld-go: set crd ns default sid helloworld-go: updating resource Request entity too large: limit is 3145728
knative-serving guard-service-6cb5b4bc6-g5h8d guard-service info	Failed to update KubeApi with new config default.helloworld-go: set crd ns default sid helloworld-go: updating resource Request entity too large: limit is 3145728
knative-serving guard-service-6cb5b4bc6-g5h8d guard-service info	Failed to update KubeApi with new config default.helloworld-go: set crd ns default sid helloworld-go: updating resource Request entity too large: limit is 3145728
knative-serving guard-service-6cb5b4bc6-g5h8d guard-service info	Failed to update KubeApi with new config default.helloworld-go: set crd ns default sid helloworld-go: updating resource Request entity too large: limit is 3145728
knative-serving guard-service-6cb5b4bc6-g5h8d guard-service info	Failed to update KubeApi with new config default.helloworld-go: set crd ns default sid helloworld-go: updating resource Request entity too large: limit is 3145728
....

This happens with security-guard 0.4

[David Hadas]

Interesting.
I wonder what exploded... which field do the code fail to protect the size of...

Can you do a kubectl get guardians.guard.security.knative.dev helloworld-go -o yaml and see which parameters is exploding?

I am struggling to get this script running on macos (getting tr: Illegal byte sequence)

[Roland Huß]

I am struggling to get this script running on macos (getting tr: Illegal byte sequence)

I'm just running it from inside the cluster via kubectl run curl -itq --rm --image=k8spatterns/curl-jq --command -- sh

[Roland Huß]

It's the vals: field that appends a new entry for every random number, like in:

     ....
     qs:
       kv:
         vals:
              00BBd0wkzq0OYL64pL76:
                digits: 0
                flags: 0
                letters: 0
                nonreadables: 0
                schars: 0
                sequences: 0
                spaces: 0
                unicodeFlags: null
                unicodes: 0
              00EI8y77Jf30W4gkJNTW:
                digits: 0
                flags: 0
                letters: 0
                nonreadables: 0
                schars: 0
                sequences: 0
                spaces: 0
                unicodeFlags: null
                unicodes: 0
              00TqarsD88lk0XTrTe4r:
                digits: 0
                flags: 0

[Roland Huß]

btw, on macOS you can use: LC_ALL=C tr -dc 'a-zA-Z0-9' < /dev/urandom | head -c 10 ; echo for generating random strings.

[David Hadas]

ok, so we have hit this... https://github.com/knative-sandbox/security-guard/blob/504e9f02d03e304151e4baf08f16de7ab34daae6/pkg/apis/guard/v1alpha1/keyval.go#L169

The key value does not yet include code for generalizing from multiple keys - i.e. it does not have the ability to move from learning the specifics of each named key and what we expect from the value of such key to defining our expectation of any unnamed key. Support for manual configuration of unnamed keys exists, but the automated learning code never uses it - never generalizes from multiple keys. The planned design is to have a limit on the number of keys and once we reach this limit to start collapsing keys to form an unnamed key.

I can prioritize work on adding this ability and get at least an initial solution out quickly or we can consider the current behavior sufficient for now and fix it before we GA.