[HELP] Bouncer not blocking banned IPs
rgomezceis opened this issue · comments
Hello,
I have crowdsec installed in OPNsense machine that works as LAPI.
Also we have a crowdsec docker container (LAPI disabled) that connects to OPNsense LAPI for acquiring logs of traefik using crowdsecurity/traefik collection.
crowdsec:
container_name: crowdsec
image: crowdsecurity/crowdsec
restart: unless-stopped
environment:
TZ: Europe/Madrid
DISABLE_LOCAL_API: true
LOCAL_API_URL: "OPNSense LAPI URL"
AGENT_USERNAME: "---"
AGENT_PASSWORD: "---"
COLLECTIONS: "crowdsecurity/traefik"
GID: "${GID-1000}"
volumes:
- /mnt/docker/crowdsec/acquis.yaml:/etc/crowdsec/acquis.yaml
- traefik_logs:/var/log/traefik
- crowdsec_db_data:/var/lib/crowdsec/data/
- crowdsec_data:/etc/crowdsec/
Adding a new bouncer in OPNsense Crowdsec to get API KEY using:
cscli bouncers add traefik-bouncer
I've installed traefik bouncer plugin using this labels and then I've attached the middleware to the docker service:
environment:
# GET REAL IP FROM CLOUDFLARE
- TRAEFIK_ENTRYPOINTS_WEBSECURE_PROXYPROTOCOL_TRUSTEDIPS=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22
- TRAEFIK_ENTRYPOINTS_WEBSECURE_FORWARDEDHEADERS_TRUSTEDIPS=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22
# ADD Traefik Bouncer Plugin
- TRAEFIK_EXPERIMENTAL_PLUGINS_BOUNCER_MODULENAME=github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin
- TRAEFIK_EXPERIMENTAL_PLUGINS_BOUNCER_VERSION=v1.1.16
labels:
traefik.http.middlewares.crowdsec.plugin.bouncer.enabled: true
traefik.http.middlewares.crowdsec.plugin.bouncer.loglevel: "DEBUG"
traefik.http.middlewares.crowdsec.plugin.bouncer.crowdseclapihost: "OPNsense LAPI host"
traefik.http.middlewares.crowdsec.plugin.bouncer.crowdseclapikey: "API KEY GENERATED IN OPNSense LAPI"
traefik.http.middlewares.crowdsec.plugin.bouncer.crowdsecmode: "stream"
traefik.http.middlewares.crowdsec.plugin.bouncer.forwardedheaderstrustedips: "173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22" # CLOUDFLARE IPS
So apparently works but not blocking clients when accessing to the service...
traefik-bouncer 192.168.1.4 ✔️ 2023-10-19T10:25:52Z Go-http-client 1.1 api-key
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 12:02:52 handleStreamCache:updated
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 12:02:52 New initialized mode:stream
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 12:22:52 cache:SetDecision ip:31.4.140.XXX isBanned:true duration:14357s
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 12:22:52 handleStreamCache:updated
What are you expecting the bouncer to do?
Cause it a web server mitigation it is intended to return a unauthorized response code, it will not completely block the user from the server you would have to inject the rule into your OPNSense if that is your router firewall.
What are you expecting the bouncer to do?
Cause it a web server mitigation it is intended to return a unauthorized response code, it will not completely block the user from the server you would have to inject the rule into your OPNSense if that is your router firewall.
I've expect that traefik block the web request to the service... but not working.
What are you expecting the bouncer to do?
Cause it a web server mitigation it is intended to return a unauthorized response code, it will not completely block the user from the server you would have to inject the rule into your OPNSense if that is your router firewall.I've expect that traefik block the web request to the service... but not working.
Okay, have you added a temporary decision to your WAN IP to see what response you get?
sudo cscli decisions add --ip <your_wan> -d 1m
curl <your_site> -vv
Will add a ban for 1 minute, make sure the curl request doesnt come from your internal IP
if you dont want to add a decision on your WAN then you can use a VPN or rent a vps for couple of minutes.
What are you expecting the bouncer to do?
Cause it a web server mitigation it is intended to return a unauthorized response code, it will not completely block the user from the server you would have to inject the rule into your OPNSense if that is your router firewall.I've expect that traefik block the web request to the service... but not working.
Okay, have you added a temporary decision to your WAN IP to see what response you get?
sudo cscli decisions add --ip <your_wan> -d 1m curl <your_site> -vv
Will add a ban for 1 minute, make sure the curl request doesnt come from your internal IP
Yes I've added it.
Other bouncers blocks it, but traefik bouncer does nothing
Our infraestructure is: Cloudflare -> Traefik -> Web Server
How the bouncer works? How it blocks the IPs?
What was the response from traefik?
How the bouncer works? How it blocks the IPs?
Cause it a web server mitigation it is intended to return a unauthorized response code,
Pasting the curl output minus your server fqdn can help us debug
Adding logs when you send the request might help. You've only shown the moment when the "cron" adds the banned IP to its own storage.
What was the response from traefik?
How the bouncer works? How it blocks the IPs?
Cause it a web server mitigation it is intended to return a unauthorized response code,
Pasting the curl output minus your server fqdn can help us debug
I'm going to do a webrequest to traefik service without passing from cloudflare.
I've added a ban decision to my local ip
cscli decisions add --ip 172.16.32.2 -d 1m
Now others bouncers block my local ip, but traefik nothing.. here logs:
C:\Users\rgomez>curl -vv https://xxxx.xxxx.xxx
* Trying 192.168.1.4:443...
* Connected to xxxx.xxxx.xxx (192.168.1.4) port 443 (#0)
* schannel: disabled automatic use of client certificate
* ALPN: offers http/1.1
* ALPN: server accepted http/1.1
* using HTTP/1.1
> GET / HTTP/1.1
> Host: xxxxx.xxx.xxx
> User-Agent: curl/8.0.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Content-Length: 867
< Content-Type: text/html
< Date: Thu, 19 Oct 2023 10:49:20 GMT
< Etag: "652fba9b-363"
< Last-Modified: Wed, 18 Oct 2023 10:59:39 GMT
< Server: nginx/1.25.1
And accesing to other site using firewall-bouncer:
C:\Users\rgomez>curl -vv https://xxxx.xxxx.xxxx
* Trying 10.0.0.4:443...
* connect to 10.0.0.4 port 443 failed: Timed out
* Failed to connect to www.ceis.es port 443 after 21048 ms: Couldn't connect to server
* Closing connection 0
curl: (28) Failed to connect xxx.xxx.xxx port 443 after 21048 ms: Couldn't connect to server
I think you don't apply the crowdsec middleware to the web service in traefik
I think you don't apply the crowdsec middleware to the web service in traefik
Yes mate, look:
traefik.http.routers.portalcliente.middlewares: "crowdsec,cors"
Adding logs when you send the request might help. You've only shown the moment when the "cron" adds the banned IP to its own storage.
I don't speak about your curl logs but traefik logs, when the plugin receive request in debug mode, it log a lot of things
Adding logs when you send the request might help. You've only shown the moment when the "cron" adds the banned IP to its own storage.
I don't speak about your curl logs but traefik logs, when the plugin receive request in debug mode, it log a lot of things
Yes but there's a lot of decisions...
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 103.21.244.0/22 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 103.22.200.0/22 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 103.31.4.0/22 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 141.101.64.0/18 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 108.162.192.0/18 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 190.93.240.0/20 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 188.114.96.0/20 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 197.234.240.0/22 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 198.41.128.0/17 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 162.158.0.0/15 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 104.16.0.0/13 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 104.24.0.0/14 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 172.64.0.0/13 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 131.0.72.0/22 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 No IP provided for ClientTrustedIPs
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 173.245.48.0/20 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 103.21.244.0/22 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 103.22.200.0/22 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 103.31.4.0/22 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 141.101.64.0/18 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 108.162.192.0/18 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 190.93.240.0/20 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 188.114.96.0/20 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 197.234.240.0/22 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 198.41.128.0/17 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 162.158.0.0/15 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 104.16.0.0/13 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 104.24.0.0/14 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 172.64.0.0/13 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 IP network 131.0.72.0/22 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 getTLSConfigCrowdsec:CrowdsecLapiScheme https:no
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 cache:New initialized isRedis:false
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 cache:GetDecision ip:updated
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 cache:SetDecision ip:updated isBanned:false duration:59s
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 cache:DeleteDecision ip:125.59.252.103
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 cache:DeleteDecision ip:197.232.18.128
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 cache:DeleteDecision ip:157.122.183.219
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 cache:DeleteDecision ip:175.6.103.113
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:43 cache:DeleteDecision ip:124.38.110.106
.....
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:50 handleStreamCache:updated
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:16:50 New initialized mode:stream
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:17:50 cache:GetDecision ip:updated
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:17:50 cache:SetDecision ip:updated isBanned:false duration:59s
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:17:51 handleStreamCache:updated
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:18:50 cache:GetDecision ip:updated
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:18:50 cache:SetDecision ip:updated isBanned:false duration:59s
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:18:51 handleStreamCache:updated
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:19:50 cache:GetDecision ip:updated
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:19:50 cache:SetDecision ip:updated isBanned:false duration:59s
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:19:51 handleStreamCache:updated
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:20:50 cache:GetDecision ip:updated
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:20:50 cache:SetDecision ip:updated isBanned:false duration:59s
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:20:51 handleStreamCache:updated
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:21:50 cache:GetDecision ip:updated
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:21:50 cache:SetDecision ip:updated isBanned:false duration:59s
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:21:51 handleStreamCache:updated
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:22:50 cache:GetDecision ip:updated
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:22:50 cache:SetDecision ip:updated isBanned:false duration:59s
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:22:51 handleStreamCache:updated
There's no entry like this:
Jan 30 09:39:45 traefik traefik[1757]: DEBUG: CrowdsecBouncerTraefikPlugin: 2023/01/30 09:39:45 ServeHTTP ip:10.10.100.38 isTrusted:false
Jan 30 09:39:45 traefik traefik[1757]: DEBUG: CrowdsecBouncerTraefikPlugin: 2023/01/30 09:39:45 ServeHTTP:handleNoStreamCache ip:10.10.100.38 isBanned:true handleNoStreamCache:banned
If I use 'live' mode there isn't any query to LAPI (time not change):
traefik-bouncer 192.168.1.4 ✔️ 2023-10-19T13:27:50Z Go-http-client 1.1 api-key
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 173.245.48.0/20 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 103.21.244.0/22 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 103.22.200.0/22 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 103.31.4.0/22 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 141.101.64.0/18 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 108.162.192.0/18 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 190.93.240.0/20 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 188.114.96.0/20 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 197.234.240.0/22 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 198.41.128.0/17 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 162.158.0.0/15 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 104.16.0.0/13 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 104.24.0.0/14 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 172.64.0.0/13 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 131.0.72.0/22 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 No IP provided for ClientTrustedIPs
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 173.245.48.0/20 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 103.21.244.0/22 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 103.22.200.0/22 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 103.31.4.0/22 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 141.101.64.0/18 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 108.162.192.0/18 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 190.93.240.0/20 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 188.114.96.0/20 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 197.234.240.0/22 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 198.41.128.0/17 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 162.158.0.0/15 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 104.16.0.0/13 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 104.24.0.0/14 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 172.64.0.0/13 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 IP network 131.0.72.0/22 is trusted
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 getTLSConfigCrowdsec:CrowdsecLapiScheme https:no
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 cache:New initialized isRedis:false
DEBUG: CrowdsecBouncerTraefikPlugin: 2023/10/19 13:28:22 New initialized mode:live
The middleware is not used. Maybe show us your docker compose config where you use
traefik.http.routers.portalcliente.middlewares: "crowdsec,cors"
The middleware is not used. Maybe show us your docker compose config where you use
traefik.http.routers.portalcliente.middlewares: "crowdsec,cors"
I've definded the middleware in traefik docker compose and referenced it in the service...
Traefik compose:
labels:
traefik.http.middlewares.crowdsec.plugin.bouncer.enabled: true
traefik.http.middlewares.crowdsec.plugin.bouncer.loglevel: "DEBUG"
traefik.http.middlewares.crowdsec.plugin.bouncer.crowdseclapihost: "OPNsense LAPI host"
traefik.http.middlewares.crowdsec.plugin.bouncer.crowdseclapikey: "API KEY GENERATED IN OPNSense LAPI"
traefik.http.middlewares.crowdsec.plugin.bouncer.crowdsecmode: "stream"
traefik.http.middlewares.crowdsec.plugin.bouncer.forwardedheaderstrustedips: "173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22" # CLOUDFLARE IPS
Service compose:
version: "3.8"
services:
portalcliente-test:
container_name:
image:
restart: unless-stopped
networks:
public:
labels:
traefik.enable: true
traefik.http.routers.portalcliente_test.rule: "Host(`xxx.xxxx.xxxx`)"
traefik.http.routers.portalcliente_test.entrypoints: "web,websecure"
traefik.http.routers.portalcliente_test.tls: true
traefik.http.routers.portalcliente.middlewares: "crowdsec,cors" # HERE
traefik.http.services.portalcliente_test.loadbalancer.server.port: 80
traefik.http.routers.portalcliente_test.service: "portalcliente_test"
# AUTO UPDATE IMAGE
com.centurylinklabs.watchtower.enable: true
networks:
public:
external: true
traefik.http.routers.portalcliente.middlewares: "crowdsec,cors"
=>
traefik.http.routers.portalcliente_test.middlewares: "crowdsec,cors"
?
traefik.http.routers.portalcliente.middlewares: "crowdsec,cors" => traefik.http.routers.portalcliente_test.middlewares: "crowdsec,cors" ?
OMG, my bad. Thx :(
Don't forget to put a star, that's help us a lot 👍