fail to send HTTP request to modsec

[igoooor]

Hello there,

I have notice that very randomly, traefik is not able to call the modsec plugin, even though it works most of the time. Honestly I'm not really sure what could trigger that.
Here is the traefik log that shows the issue:

fail to send HTTP request to modsec: Get "http://owasp-modsec.default.svc.cluster.local:80/de": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

I did a load test (with very little users, just to have a constant load to try to trigger that error), and the load test was going well, the modsec was doing it's job correctly, but for a duration of 2 seconds in the middle of the loadtest, these errors appeared.

I searched a bit on traefik issues if I could find anything related to that, and all I could find in a couple issues is that:

the error context deadline exceeded comes because your plugin takes more than 5s to load.

Also, during the duration of "failure", modsec container does not show any log, which also shows that traffic is not reaching to it.
The problem does not only appear during loadtest but also randomly during the day with "normal" requests.

Would you have any clue what could be causing that? I hope it's the right place to ask.

[Alexis Couvreur]

Maybe the modsec container died ?
I believe the plugin is working correctly but for so;e reason the modsec container is not able to answer the load.

You should try to monitor the modsec container load.

Can you share the results and maybe your deployment file ?

Thank you

[igoooor]

I did monitor the modsec container and it did not restart.
Here is my deployment and config:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: owasp-modsec
spec:
  selector:
    matchLabels:
      run: owasp-modsec
  replicas: 5
  template:
    metadata:
      labels:
        run: owasp-modsec
    spec:
      containers:
        - name: owasp-modsec
          image: owasp/modsecurity-crs:3.3.2-apache-alpine
          securityContext:
            allowPrivilegeEscalation: false
          ports:
            - containerPort: 80
              name: http
          env:
            - name: PARANOIA
              value: "1"
            - name: ANOMALY_INBOUND
              value: "10"
            - name: ANOMALY_OUTBOUND
              value: "5"
            - name: BACKEND
              value: http://dummy-owasp:80
            - name: MAX_FILE_SIZE
              value: "1000000000"
            - name: COMBINED_FILE_SIZES
              value: "1000000000"
            - name: MODSEC_REQ_BODY_LIMIT
              value: "1000000000"
          volumeMounts:
            - mountPath: /etc/modsecurity.d/owasp-crs/rules/REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf
              name: owasp-modsec-rules-before
              subPath: REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf
      volumes:
        - name: owasp-modsec-rules-before
          configMap:
            name: owasp-modsec-900-exclusion
            items:
              - key: REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf
                path: REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf
---
apiVersion: v1
data:
  REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf: |
    SecRuleRemoveById 200004
kind: ConfigMap
metadata:
  name: owasp-modsec-900-exclusion
  namespace: default
---
apiVersion: v1
kind: Service
metadata:
  name: owasp-modsec
  labels:
    run: owasp-modsec
spec:
  type: ClusterIP
  ports:
    - port: 80
      name: http
      protocol: TCP
      targetPort: 80
  selector:
    run: owasp-modsec

[Alexis Couvreur]

Maybe it did not restart but was exhausted ?

Do you have a CPU and Memory graph of the container during the execution ?

[igoooor]

Both traefik and modsec containers have a flat cpu and memory utilization (meaning no spike or anything)

[Alexis Couvreur]

And can you reproduce the issue every time ? Or does it seem random ?

[igoooor]

it is totally random. For example now it just happened when my uptimerobot called the endpoint, but I opened the same page just after and it was working again.

[igoooor]

I opened a tunnel to the web port of the owasp container and never ran into an issue, it seems that it only happens when the call if made from the plugin (i.e. from the traefik pod)

[igoooor]

I don't want to jinx it, but I may have found the issue.
I initially had the url configured like this:
modSecurityUrl=http://owasp-modsec.default.svc.cluster.local
which I now changed to
modSecurityUrl=http://owasp-modsec (given that they are in the same namespace)
And since, I don't have errors.
I will monitor it, and close the issue if it's all good now

[Alexis Couvreur]

I don't want to jinx it, but I may have found the issue. I initially had the url configured like this: modSecurityUrl=http://owasp-modsec.default.svc.cluster.local which I now changed to modSecurityUrl=http://owasp-modsec (given that they are in the same namespace) And since, I don't have errors. I will monitor it, and close the issue if it's all good now

That's nice to hear, even though the issue is kinda weird. I believe it should work with the svc url...

[igoooor]

I mark this one as resolved, I don't understand either why it would fail with the svc url but well...