Giters

NginxProxyManager / nginx-proxy-manager

Docker container for managing Nginx proxy hosts with a simple, powerful interface

Home Page:https://nginxproxymanager.com

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Integration with crowdsecurity/cs-nginx-bouncer

Sparkxxx opened this issue · comments

commented

Is your feature request related to a problem? Please describe.
I would like to add protection for sites to NPM and be able to block IP's based on reputation/scenario using Crowdsecurity https://github.com/crowdsecurity which is a log analyzer and reactive firewall, something like fail2ban but in a much modern architecture and suitable for the containers world and having a distributed banlist generated from users around the world.

The question/feature request is how/where to add the https://github.com/crowdsecurity/cs-nginx-bouncer module to nginx config.
This bouncer leverages nginx lua's API, namely access_by_lua_file.
New/unknown IPs are checked against crowdsec API, and if request should be blocked, a 403 is returned to the user, and put in cache. The exact desired action can be configured in Crowdsec, like displaying captcas etc.

Describe the solution you'd like
I would like to have an easy way (example maybe) to deploy the bouncer without breaking things in npm.

Describe alternatives you've considered
None so far since it might take you minutes to offer a solution and it would take me hours to tweak things and break them :)

Additional context
Please have a look at https://crowdsec.net/ to understand what is crowdsec and how it works.
I use jc21/nginx-proxy-manager:latest for NPM and https://github.com/crowdsecurity/crowdsec/tree/master/docker for Crowdsec containers analyzing the nginx logs allready written by NPM on disk (docker map volumes ./npm_data/logs/:/var/log/nginx/ and have a quick look at acquis.yaml which tells crowdsec which logs to parse, mine looks like this):
`filenames:

  • /var/log/nginx/*.log
    labels:
    type: nginx`

Thank you and hope you'd consider this as a valuable addition to NPM

commented

Referencing #39.

commented

I have read that feature request but, let me justify better my request.
As far as I know fail2ban works only on one machine so it would have to manipulate some iptables/ipsets or scripts to work over a network and this implies many moving parts. It is basically a stand alone application from before the container world and I'm not even speaking here about swarms. I know it's usefull, I'm using it but not in the container world where in my opinion it looks like a dinosaur.
Crowdsec (runs on docker and has 3.3K stars on github) on the other hand is a more modern and advanced way of achieving the same goals as with fail2ban but in a distributed infrastructure with networking, VM's, containers, barebones, detecting atacks, redirecting, banning, serving captchas, etc., with a graphical interface and we love the GUIs. Bouncers and detectors don't even have to be on the same machine/container.
Have a look at their agents/bouncers HUB https://hub.crowdsec.net/ and BLOG https://crowdsec.net/blog/ for what is already available.

My requested integration ( https://hub.crowdsec.net/author/crowdsecurity/bouncers/cs-nginx-bouncer ) is already a lua script that works in OpenResty, so it would only be, in my opinion, a matter of copy-paste in the config file of NPM and not writing a new module or script to integrate it.

Please allow me to cite from Crowdsec's website:

  • CrowdSec is an open-source and collaborative EDR. Analyze behaviors, respond to attacks & share signals across the community.

  • Crowdsec-agent is an open-source and lightweight software that allows you to detect peers with malevolent behaviors and block them from accessing your systems at various level (infrastructural, system, applicative).

  • To achieve this, Crowdsec-agent reads logs from different sources (files, streams ...) to parse, normalize and enrich them before matching them to threats patterns called scenarios.

  • Crowdsec-agent is a modular and plug-able framework, it ships a large variety of well known popular scenarios; users can choose what scenarios they want to be protected from as well as easily adding new custom ones to better fit their environment.

  • Detected malevolent peers can then be prevented from accessing your resources by deploying bouncers at various levels (applicative, system, infrastructural) of your stack.

  • One of the advantages of Crowdsec when compared to other solutions is its crowd-sourced aspect : Meta information about detected attacks (source IP, time and triggered scenario) are sent to a central API and then shared amongst all users.

  • Thanks to this, besides detecting and stopping attacks in real time based on your logs, it allows you to preemptively block known bad actors from accessing your information system.

I understand that Crowdsec is a new tool for many but it is able to do even more than fail2ban and the learning curve is low and in the long run it would overtake f2b and all you have to do is configure it where to read the logs and it is already packed with detectors and ready to take action trough bouncers on new detected atacker or the ones downloaded from the community list.
It is a set and forget application that can be further integrated with others, being prometheus or your own application over the network trough it's api.

commented

Yeah, it was not my intention to say this issue was the same thing as the fail2ban one, I just wanted to cross-link these issues, since this they seem mutually exclusive.

commented

I didn't thought you were saying that, just wanted to clarify a little bit since with appearance of crowdsec (v0.0.1 - May 15, 2020), supporting an old tool and reinventing the wheel and wasting time building around f2b seeamd like a bad idea.
Hope you like my proposal and looking forward to see it integrated in NPM.

commented

Hello, I'll add my vote for this feature. I thinks this is very different from fail2ban in which follows the crowdsourcing approach.

Thanks!

commented

@Sparkxxx do you have the manual steps to do this for the time being? I'm not very familiar with crowdsec so forgive my naivete, I think I have setup crowdsec and metabse in the same dockerfile using the guide here and here, but when i test with nikto from another machine on the local network I do not see a new decision and logs don't appear to show traffic. There isn't an official docker image for the nginx bouncer either, but based on your comment it seems like there may not be a need for an extra bouncer container. Here's the setup I have so far:

version: '3'
services:
  app:
    image: 'jc21/nginx-proxy-manager:latest'
    restart: unless-stopped
    ports:
      - '80:80'
      - '81:81'
      - '443:443'
    environment:
      DB_MYSQL_HOST: "db"
      DB_MYSQL_PORT: 3306
      DB_MYSQL_USER: "npm"
      DB_MYSQL_PASSWORD: "npm"
      DB_MYSQL_NAME: "npm"
    volumes:
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt
      - logs:/var/log/nginx
  db:
    image: 'jc21/mariadb-aria:latest'
    restart: unless-stopped
    environment:
      MYSQL_ROOT_PASSWORD: 'npm'
      MYSQL_DATABASE: 'npm'
      MYSQL_USER: 'npm'
      MYSQL_PASSWORD: 'npm'
    volumes:
      - ./data/mysql:/var/lib/mysql


  #crowdsec : it will be fed nginx's logs
  #and later we're going to plug a firewall bouncer to it
  crowdsec:
    image: crowdsecurity/crowdsec:latest
    restart: always
    environment:
      #this is the list of collections we want to install
      #https://hub.crowdsec.net/author/crowdsecurity/collections/nginx
      COLLECTIONS: "crowdsecurity/nginx"
      GID: "${GID-1000}"
    depends_on:
      - 'app'
    volumes:
      - ./crowdsec/acquis.yaml:/etc/crowdsec/acquis.yaml
      - logs:/var/log/nginx
      - crowdsec-db:/var/lib/crowdsec/data/
      - crowdsec-config:/etc/crowdsec/
  
  #metabase, because security is cool, but dashboards are cooler
  dashboard:
    #we're using a custom Dockerfile so that metabase pops with pre-configured dashboards
    build: ./crowdsec/dashboard
    restart: always
    ports:
      - 3000:3000
    environment:
      MB_DB_FILE: /data/metabase.db
      MGID: "${GID-1000}"
    depends_on:
      - 'crowdsec'
    volumes:
      - crowdsec-db:/metabase-data/

volumes:
  logs:
  crowdsec-db:
  crowdsec-config:
commented

I am head of community at CrowdSec (and an avid user myself). I think it's a great idea to add support for CrowdSec as well. Also I would like to offer my assistance and a way into the CrowdSec dev team if you need any help implementing it.

I would also like to point any technical questions regarding CrowdSec to our discourse where all devs are available to help out.

@nisargjoshi95 Did you ever get this working? If not, feel free to ask on the discourse :-)

commented

@nisargjoshi95 Did you ever get this working? If not, feel free to ask on the discourse :-)

commented

@Sparkxxx please give me a buzz on the discourse, on twitter @klausagnoletti or send me a mail klaus@crowdsec.net. I'd like to ask about your experience with CrowdSec etc.

commented

I am here to cast my vote, I haven't started looking in depth yet but does anyone have crowdsec setup with the bouncer and NPM?

@klausagnoletti - Should I just go to the discourse and ask there or are there any crowdsec docs on how to set it all up while utilizing NPM? Thanks.

commented

@nisargjoshi95

I'll think your docker-compose wont work because nginx proxymanager is storing it's logs in other than the default nginx folders.

change your acquis.yml to - /var/log/nginx/*.log and then try this docker-compose: it reads the logs folder which is mounted by nginx-proxymananger container. it will read the logs but the parsing isn't that good


services:
 #crowdsec : it will be fed nginx's logs
 #and later we're going to plug a firewall bouncer to it
 crowdsec:
   image: crowdsecurity/crowdsec:v1.2.1
   restart: always
   environment:
     #this is the list of collections we want to install
     #https://hub.crowdsec.net/author/crowdsecurity/collections/nginx
     COLLECTIONS: "crowdsecurity/nginx"
     GID: "${GID-1000}"
  
   volumes:
     - ./crowdsec/acquis.yaml:/etc/crowdsec/acquis.yaml
     - /fullpath-to-nginx-proxymanager/data/logs:/var/log/nginx
     - crowdsec-db:/var/lib/crowdsec/data/
     - crowdsec-config:/etc/crowdsec/
   networks:
     crowdsec_test:
       ipv4_address: 172.20.0.4
  #metabase, because security is cool, but dashboards are cooler
 dashboard:
   #we're using a custom Dockerfile so that metabase pops with pre-configured dashboards
   build: ./crowdsec/dashboard
   restart: always
   ports:
     - 3000:3000
   environment:
     MB_DB_FILE: /data/metabase.db
     MGID: "${GID-1000}"
   depends_on:
     - 'crowdsec'
   volumes:
     - crowdsec-db:/metabase-data/
   networks:
     crowdsec_test:
       ipv4_address: 172.20.0.5

volumes:
 logs:
 crowdsec-db:
 crowdsec-config:

networks:
 crowdsec_test:
   ipam:
     driver: default
     config:
       - subnet: 172.20.0.0/24




INFO[26-11-2021 04:58:54 PM] Buckets Metrics:
+--------------------------------------+---------------+-----------+--------------+--------+---------+
|                BUCKET                | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |
+--------------------------------------+---------------+-----------+--------------+--------+---------+
| crowdsecurity/http-bad-user-agent    | -             | -         |            2 |      2 |       2 |
| crowdsecurity/http-crawl-non_statics | -             | -         |           77 |     77 |      77 |
+--------------------------------------+---------------+-----------+--------------+--------+---------+
INFO[26-11-2021 04:58:54 PM] Acquisition Metrics:
+---------------------------------------------+------------+--------------+----------------+------------------------+
|                   SOURCE                    | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+---------------------------------------------+------------+--------------+----------------+------------------------+
| file:/var/log/nginx/default-host_access.log |         37 |           30 |              7 |                     32 |
| file:/var/log/nginx/fallback-access.log     |         34 | -            |             34 | -                      |
| file:/var/log/nginx/fallback_error.log      |         23 |           23 | -              |                     20 |
| file:/var/log/nginx/proxy-host-1_access.log |        216 | -            |            216 | -                      |
| file:/var/log/nginx/proxy-host-1_error.log  |         28 |           28 | -              |                     27 |
+---------------------------------------------+------------+--------------+----------------+------------------------+
INFO[26-11-2021 04:58:54 PM] Parser Metrics:
+--------------------------------+------+--------+----------+
|            PARSERS             | HITS | PARSED | UNPARSED |
+--------------------------------+------+--------+----------+
| child-crowdsecurity/http-logs  |  243 |    100 |      143 |
| child-crowdsecurity/nginx-logs |  646 |     81 |      565 |
| crowdsecurity/dateparse-enrich |   81 |     81 | -        |
| crowdsecurity/geoip-enrich     |   30 |     30 | -        |
| crowdsecurity/http-logs        |   81 |     14 |       67 |
| crowdsecurity/nginx-logs       |  338 |     81 |      257 |
| crowdsecurity/non-syslog       |  338 |    338 | -        |
| crowdsecurity/whitelists       |   81 |     81 | -        |
+--------------------------------+------+--------+----------+
INFO[26-11-2021 04:58:54 PM] Local Api Metrics:
+----------------------+--------+------+
|        ROUTE         | METHOD | HITS |
+----------------------+--------+------+
| /v1/alerts           | GET    |    1 |
| /v1/decisions/stream | GET    | 7021 |
| /v1/watchers/login   | POST   |    3 |
+----------------------+--------+------+
INFO[26-11-2021 04:58:54 PM] Local Api Machines Metrics:
+-----------+------------+--------+------+
|  MACHINE  |   ROUTE    | METHOD | HITS |
+-----------+------------+--------+------+
| localhost | /v1/alerts | GET    |    1 |
+-----------+------------+--------+------+
INFO[26-11-2021 04:58:54 PM] Local Api Bouncers Metrics:
+---------------------+----------------------+--------+------+
|       BOUNCER       |        ROUTE         | METHOD | HITS |
+---------------------+----------------------+--------+------+
| HostFirewallBouncer | /v1/decisions/stream | GET    | 7021 |
+---------------------+----------------------+--------+------+
commented

@baudneo did this work for you? I would have thought there would be issues with NPM's custom log format.

commented

@klausagnoletti - Should I just go to the discourse and ask there or are there any crowdsec docs on how to set it all up while utilizing NPM? Thanks.

Sorry for my late reply. I don't get notifications from Github even though I enabled it :-/

Just go to the Discorse and ask. To my knowledge noone has tried integrating with npm yet.

commented

@jakern, it is a strange situation. If NPM is already running, proxying requests and I start up crowdsec with the newest config that @2Wanderer posted, everything is ok. When I reboot, NPM no longer will come up, citing an error binding to port :80 and :443 due to the crowdsec nginx bouncer using port :80 or :443. I am playing around trying to get things to mesh properly. Once I get that part stable I will start trying to integrated the logs structure.

commented

I tried creating a docker container extending the npm container and installing crowdsec-nginx-bouncer. It didn't work. Turns out npm doesn't use nginx but openresty, wghich we don't (yet) support. But it's on the roadmap. So I guess it'a a bit of a PITA to get working before then.

commented

Can you give a rough estimation when this will probably happen?

commented

No, it is not scheduled. Maybe the community will contribute it. It's open source after all :-)

commented

I already tried it with a custom dockerfile:

FROM jc21/nginx-proxy-manager:latest

RUN curl -s https://packagecloud.io/install/repositories/crowdsec/crowdsec/script.deb.sh | bash
RUN apt install -y crowdsec crowdsec-nginx-bouncer
RUN sed -i '1s/^/filenames:\n - \/data\/logs\/*.log\nlabels:\n  type:nginx/' /etc/crowdsec/acquis.yaml

but I get some weird errors. here are some of these:

Unpacking crowdsec-nginx-bouncer (0.0.7) ...
Setting up lua-logging (1.3.0-1) ...
Setting up libxpm4:amd64 (1:3.5.12-1) ...
Setting up nginx-common (1.14.2-2+deb10u4) ...

Configuration file '/etc/nginx/mime.types'
 ==> File on system created by you or by a script.
 ==> File also in package provided by package maintainer.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** mime.types (Y/I/N/O/D/Z) [default=N] ? dpkg: error processing package nginx-common (--configure):
 end of file on stdin at conffile prompt
Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-4) ...
dpkg: dependency problems prevent configuration of nginx-full:
 nginx-full depends on nginx-common (= 1.14.2-2+deb10u4); however:
  Package nginx-common is not configured yet.

dpkg: error processing package nginx-full (--configure):
 dependency problems - leaving unconfigured
Setting up libjbig0:amd64 (2.1-3.1+b2) ...
Setting up libicu63:amd64 (63.1-6+deb10u2) ...
Setting up lua-sql-sqlite3:amd64 (2.3.4-1+b1) ...
dpkg: dependency problems prevent configuration of libnginx-mod-http-xslt-filter:
 libnginx-mod-http-xslt-filter depends on nginx-common (= 1.14.2-2+deb10u4); however:
  Package nginx-common is not configured yet.

dpkg: error processing package libnginx-mod-http-xslt-filter (--configure):
 dependency problems - leaving unconfigured
Setting up libjpeg62-turbo:amd64 (1:1.5.2-2+deb10u1) ...
dpkg: dependency problems prevent configuration of libnginx-mod-http-auth-pam:
 libnginx-mod-http-auth-pam depends on nginx-common (= 1.14.2-2+deb10u4); however:
  Package nginx-common is not configured yet.

dpkg: error processing package libnginx-mod-http-auth-pam (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libnginx-mod-http-geoip:
 libnginx-mod-http-geoip depends on nginx-common (= 1.14.2-2+deb10u4); however:
  Package nginx-common is not configured yet.

dpkg: error processing package libnginx-mod-http-geoip (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libnginx-mod-http-ndk:
 libnginx-mod-http-ndk depends on nginx-common (= 1.14.2-2+deb10u4); however:
  Package nginx-common is not configured yet.

maybe you can help? or is this the error you mentioned, that not the real nginx but openresty is used?

commented

This is the error I mentioned. Openresty != nginx (and they're not compatible). So the .deb complains that dependencies are not installed. So it won't work, unfortunately.

commented

Anyways I have good news @leon1995. I asked the developers just to be sure. And it turns out they're working on it as we speak. Only a new bouncer is needed as logfiles are the same, apparently. I'll ping you here once it's out in some usable shape or form. Are you willing to do a beta test?

commented

@klausagnoletti I'm running on unRaid with NPM and willing to beta test if you need more people. I have to be honest: I have a pretty private server, so there won't be many visitors. I can imagine you'd rather have a beta tester with some spicey logs.

commented

I can imagine you'd rather have a beta tester with some spicey logs.

Not gonna lie: That would be more interesting. But it won't prevent you to join the fun :-)

commented

@klausagnoletti I would also join the beta. However, mine is also a private server with multiple services and just a few people using it (maybe up to ten I think)

commented

So... @leon1995 and I will visit each other's servers like crazy til steam's coming out of our logs. Will that work for you, @klausagnoletti? 🤪

commented

Sure, hit me up at klaus (at) crowdsec (dot) net. You'd have to do a Docker container on your own (but would be happy to help :-)

commented

I sent you an email

commented

I can help testing if you wish. I have a host with Dietpi for x86, several dockers containers, one with Nginx proxy manager, other with crowdsec with the logs from Nginx proxy manager logs mapped, and also the firewall bouncer installed on the host and connected to Crowdsec's docker container.
I have installed Crowdsec last week, so my knowledge about it is very limited.

commented

I can help testing if you wish. I have a host with Dietpi for x86, several dockers containers, one with Nginx proxy manager, other with crowdsec with the logs from Nginx proxy manager logs mapped, and also the firewall bouncer installed on the host and connected to Crowdsec's docker container. I have installed Crowdsec last week, so my knowledge about it is very limited.

Cool, please send me an email so I have your information in my inbox :-)

commented

Now there's interesting news! @dhernan3 @leon1995 @sanderdatema @baudneo @2Wanderer @nisargjoshi95 @Sparkxxx

https://docs.crowdsec.net/docs/bouncers/openresty/

The bouncer is out. It should be easy to extend the existing npm Dockerfile with that since it's available as a .deb and npm is based on Debian.
For log parsing and scenarios the normal nginx collection can be used. Contrary to my former believe, the log format is indeed the same. Find it here: https://hub.crowdsec.net/author/crowdsecurity/collections/nginx

Let me know what comes out of experimenting with it :-)

commented

Would be great if anyone can get this working and write some quick instructions :)

  • I might give it at go myself when I have the time.
commented

Actually there's many ways to use it. The quick and dirty way is to have the agent and firewall installed on the host (which I have on my own instance as of a couple of days), point to the log files in the /etc/crowdsec/aquis.yaml (of course adjusted to your environment):

#NPM - Openresty logs
filenames:
  - /home/klaus/containers/npm/data/logs/*.log
    #  - /home/klaus/containers/npm/data/logs/proxy-host-2_access.log
    #  - /home/klaus/containers/npm/data/logs/proxy-host-3_access.log
labels:
  type: nginx
---

Protip: remember the --- as a separator between entries or you get weird errors :-)

After this, install the nginx collection: sudo cscli collections install crowdsecurity/nginx and reload crowdsec sudo systemctl reload crowdsec.

Another approach would be to extend the npm container with the new openresty bouncer and run the CrowdSec agent either on the docker host or in its own container (and maybe share data between the containers using docker volumes. This is not yet tested but knock yourselves out :-)

Let me know what comes out of it!

commented

Actually there's many ways to use it. The quick and dirty way is to have the agent and firewall installed on the host (which I have on my own instance as of a couple of days), point to the log files in the /etc/crowdsec/aquis.yaml (of course adjusted to your environment):

Let me know what comes out of it!

Perfect, thanks - I'll give it at go as soon as i get the change :)

  • I'll probaly try using the docker container, seeing as pretty much everything else in my setup is containerized.
commented
  • I'll probaly try using the docker container, seeing as pretty much everything else in my setup is containerized.

You're most welcome - only issue with that approach is that you'd have to either containerize the openresty bouncer or the firewall bouncer (which would be messy since it would need root permissions) or install just the firewall bouncer on the host and find a way for the CrowdSec container to talk to it in order to be able to block anything. Feel free to post at https://discourse.crowdsec.net if you run into problems.

commented

Hey!
So there has been a recent development in the case. We have modificed log parsers and have a PR ready:
crowdsecurity/hub#358

The big question - and what we need your help for - is input to how this should be integrated with NPM to be as usable to the NPM community as possible. If you have good suggestions wither write them here or in our Discord (in #dev-general)

Also I added an issue on logging Admin panel by default. Currently it's disabled which makes it hard (e.g. impossible for CrowdSec or anything else to protect it): #1800

Let me know what you think!

commented

I have been trying and trying to get the openresty bouncer to work with no success. All my other bouncers work fine except for openresty. I added the LOG_FILE directive to the openresty bouncer file and the log file is never created. cscli bouncers list shows this

`

NAME IP ADDRESS VALID LAST API PULL TYPE VERSION

FirewallBouncer-1643162803 ::1 ✔️ 2022-01-26T02:40:25Z crowdsec-firewall-bouncer v0.0.22-debian-pragmatic-f64e94b59a948717c3dc848f9abebb27b5974714
zm-iptables 10.0.0.30 ✔️ 2022-01-26T02:40:22Z crowdsec-firewall-bouncer v0.0.21-debian-pragmatic-eebd0b02ac6bb8cb330b3fcacc1f400595fbc6f7
local-cloudflare ::1 ✔️ 2022-01-26T02:40:30Z crowdsec-cloudflare-bouncer v0.0.8-debian-pragmatic-38768ad6e47bc7ce058668ef286de303f897d705
local-openresty ✔️ 2022-01-26T02:13:59Z
---------------------------------------------------------------------------------------------------------------------------------------------------------------------`

openresty and openresty-opm installed by the openresty bouncer do not have logging enabled and I am uncertain how to enable it properly as the openresty docs only point towards nginx docs.

I cant debug the bouncer without logging, can someone point me in the right direction to enable logging in openresty so that the bouncer can attempt to output log files for debug?

Can anyone post a working non-docker config where the openresty bouncer is actually working and confirm, that someone has the openresty bouncer actually working?

Edit: Also it seems the nginx collection and its nginx parser do not accept the NPM logs

NPM log message:

[26/Jan/2022:03:27:25 +0000] - 200 200 - GET https xxx.EXAMPLE.com \"/zm/index.php?view=login\" [Client IPADDRESS] [Length 2321] [Gzip -] [Sent-to 10.0.0.30] \"Dalvik/2.1.0 (Linux; U; Android 11; SM-N975W Build/RP1A.200720.012)\" \"-\"

nginx parser patterns

pattern: '(%{IPORHOST:target_fqdn} )?%{IPORHOST:remote_addr} - (%{NGUSER:remote_user})? \[%{HTTPDATE:time_local}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status} %{NUMBER:body_bytes_sent} "%{NOTDQUOTE:http_referer}" "%{NOTDQUOTE:http_user_agent}"( %{NUMBER:request_length} %{NUMBER:request_time} \[%{DATA:proxy_upstream_name}\] \[%{DATA:proxy_alternative_upstream_name}\])?'

pattern: '(%{IPORHOST:target_fqdn} )?%{NGINXERRTIME:time} \[%{LOGLEVEL:loglevel}\] %{NONNEGINT:pid}#%{NONNEGINT:tid}: (\*%{NONNEGINT:cid} )?%{GREEDYDATA:message}, client: %{IPORHOST:remote_addr}, server: %{DATA:target_fqdn}, request: "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}", host: "%{IPORHOST}"'

custom crowdsec GROK patterns:

NGINXERRTIME %{YEAR}/%{MONTHNUM2}/%{DAY2} %{HOUR}:%{MINUTE}:%{SECOND}
DAY2 \d{2}

IT APPEARS THE 'MONTHNUM2' PATTERN IS MISSING!

grep -R MONTHNUM2 /etc/crowdsec
/etc/crowdsec/patterns/nginx:#NGINXERRTIME %{YEAR:year}/%{MONTHNUM2:month}/%{DAY2:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second}
/etc/crowdsec/patterns/nginx:NGINXERRTIME %{YEAR}/%{MONTHNUM2}/%{DAY2} %{HOUR}:%{MINUTE}:%{SECOND}

I will write a parser for the NPM format, but it would be great if someone could confirm they have the openresty bouncer working.

commented

I have been trying and trying to get the openresty bouncer to work with no success. All my other bouncers work fine except for openresty. I added the LOG_FILE directive to the openresty bouncer file and the log file is never created. cscli bouncers list shows this

`

NAME IP ADDRESS VALID LAST API PULL TYPE VERSION

FirewallBouncer-1643162803 ::1 ✔️ 2022-01-26T02:40:25Z crowdsec-firewall-bouncer v0.0.22-debian-pragmatic-f64e94b59a948717c3dc848f9abebb27b5974714 zm-iptables 10.0.0.30 ✔️ 2022-01-26T02:40:22Z crowdsec-firewall-bouncer v0.0.21-debian-pragmatic-eebd0b02ac6bb8cb330b3fcacc1f400595fbc6f7 local-cloudflare ::1 ✔️ 2022-01-26T02:40:30Z crowdsec-cloudflare-bouncer v0.0.8-debian-pragmatic-38768ad6e47bc7ce058668ef286de303f897d705 local-openresty ✔️ 2022-01-26T02:13:59Z ---------------------------------------------------------------------------------------------------------------------------------------------------------------------`

openresty and openresty-opm installed by the openresty bouncer do not have logging enabled and I am uncertain how to enable it properly as the openresty docs only point towards nginx docs.

I cant debug the bouncer without logging, can someone point me in the right direction to enable logging in openresty so that the bouncer can attempt to output log files for debug?

Can anyone post a working non-docker config where the openresty bouncer is actually working and confirm, that someone has the openresty bouncer actually working?

Edit: Also it seems the nginx collection and its nginx parser do not accept the NPM logs

NPM log message:

[26/Jan/2022:03:27:25 +0000] - 200 200 - GET https xxx.EXAMPLE.com \"/zm/index.php?view=login\" [Client IPADDRESS] [Length 2321] [Gzip -] [Sent-to 10.0.0.30] \"Dalvik/2.1.0 (Linux; U; Android 11; SM-N975W Build/RP1A.200720.012)\" \"-\"

nginx parser patterns

pattern: '(%{IPORHOST:target_fqdn} )?%{IPORHOST:remote_addr} - (%{NGUSER:remote_user})? \[%{HTTPDATE:time_local}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status} %{NUMBER:body_bytes_sent} "%{NOTDQUOTE:http_referer}" "%{NOTDQUOTE:http_user_agent}"( %{NUMBER:request_length} %{NUMBER:request_time} \[%{DATA:proxy_upstream_name}\] \[%{DATA:proxy_alternative_upstream_name}\])?'

pattern: '(%{IPORHOST:target_fqdn} )?%{NGINXERRTIME:time} \[%{LOGLEVEL:loglevel}\] %{NONNEGINT:pid}#%{NONNEGINT:tid}: (\*%{NONNEGINT:cid} )?%{GREEDYDATA:message}, client: %{IPORHOST:remote_addr}, server: %{DATA:target_fqdn}, request: "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}", host: "%{IPORHOST}"'

custom crowdsec GROK patterns:

NGINXERRTIME %{YEAR}/%{MONTHNUM2}/%{DAY2} %{HOUR}:%{MINUTE}:%{SECOND}
DAY2 \d{2}

IT APPEARS THE 'MONTHNUM2' PATTERN IS MISSING!

grep -R MONTHNUM2 /etc/crowdsec
/etc/crowdsec/patterns/nginx:#NGINXERRTIME %{YEAR:year}/%{MONTHNUM2:month}/%{DAY2:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second}
/etc/crowdsec/patterns/nginx:NGINXERRTIME %{YEAR}/%{MONTHNUM2}/%{DAY2} %{HOUR}:%{MINUTE}:%{SECOND}

I will write a parser for the NPM format, but it would be great if someone could confirm they have the openresty bouncer working.

Did you get anywhere with this? If not, I'll arrange for you to get in touch with the dev. Ping me on Discord :-)

commented

Ok got the openresty bouncer working in jlesage/docker-nginx-proxy-manager docker image and created a pull

I'm self hosting the image currently at dockerhub.com/lepresidente/nginx-proxy-manager
which is based off my fork

You will need to set the Environment Variable CROWDSEC_BOUNCER to 1 and edit the /config/crowdsec-openresty-bouncer.conf once it has been started at least once with your API and URL to crowdsec

commented

Good job!

commented

I have forked and successfully created a docker image that incorporates openresty bouncer into NPM as it is. This means you can docker pull baudneo/nginx-proxy-manager:latest and directly replace the existing jc21/nginx_proxy_manager images. I also added the GeoIP2 with lib-MindMax DB module and modsecurity directly to openresty.

Note

  • The bouncer may not show activity right away, you need to hit a domain that is proxied by NPM for the bounder to show signs of life. YMMV.
  • Configuration file is located at /data/crowdsec/crowdsec-openresty-bouncer.conf , you will need to change the API url/key. The API_KEY and CROWDSEC_LAPI_URL ENV vars shown below may not work as expected. YMMV.
  • The bouncer is only version 0.1.0 as 0.1.1 is currently pre-release, I will upgrade to 0.1.1 soon though.

New features

  1. Enable logging for the admin dashboard, if enabled the logs will be in /data/logs/admin_panel_[access/error].log.
  2. Enable DEBUG level logging for the default openresty error.log (/data/logs/fallback_error.log) file (default level is warn).

New ENV vars

  • CROWDSEC_BOUNCER = 1 OR '1' to enable CrowdSec OpenResty bouncer.
  • CROWDSEC_LAPI = 'crowdsec local API address'
  • CROWDSEC_KEY = 'API KEY'
  • ADMIN_PANEL_LOG = 1 OR '1' to enable admin dashboard logging to logs/admin_panel_[access/error].log.
  • OPENRESTY_DEBUG = 1 OR '1' to enable DEBUG level logging for the openresty error.log (fallback_error.log) file

Forked repos:

Docker hub

Docker pull

  • docker pull baudneo/nginx-proxy-manager:cs-modsec

cscli output

  • It seems to be polling every 10 seconds
❯ cscli bouncers list
-----------------------------------------------------------------------------------------------------------------------------------------------------------
 NAME              IP ADDRESS  VALID  LAST API PULL         TYPE                         VERSION
-----------------------------------------------------------------------------------------------------------------------------------------------------------
 ZM-IPtables       10.X.X.X    ✔️      2022-02-18T22:40:54Z  crowdsec-firewall-bouncer    v0.0.22-debian-pragmatic-f64e94b59a948717c3dc848f9abebb27b5974714
 Local-cloudflare  ::1         ✔️      2022-02-18T22:40:54Z  crowdsec-cloudflare-bouncer  v0.0.8-debian-pragmatic-38768ad6e47bc7ce058668ef286de303f897d705
 Local-IPtables    ::1         ✔️      2022-02-18T22:40:56Z  crowdsec-firewall-bouncer    v0.0.22-debian-pragmatic-f64e94b59a948717c3dc848f9abebb27b5974714
 proxmox-iptables  10.X.X.X    ✔️      2022-02-18T22:41:01Z  crowdsec-firewall-bouncer    v0.0.22-debian-pragmatic-f64e94b59a948717c3dc848f9abebb27b5974714
 npm-openresty     172.18.0.3  ✔️      2022-02-18T22:40:57Z  crowdsec-openresty-bouncer   v0.0.1
-----------------------------------------------------------------------------------------------------------------------------------------------------------
❯ cscli bouncers list
-----------------------------------------------------------------------------------------------------------------------------------------------------------
 NAME              IP ADDRESS  VALID  LAST API PULL         TYPE                         VERSION
-----------------------------------------------------------------------------------------------------------------------------------------------------------
 ZM-IPtables       10.X.X.X    ✔️      2022-02-18T22:41:14Z  crowdsec-firewall-bouncer    v0.0.22-debian-pragmatic-f64e94b59a948717c3dc848f9abebb27b5974714
 Local-cloudflare  ::1         ✔️      2022-02-18T22:41:14Z  crowdsec-cloudflare-bouncer  v0.0.8-debian-pragmatic-38768ad6e47bc7ce058668ef286de303f897d705
 Local-IPtables    ::1         ✔️      2022-02-18T22:41:06Z  crowdsec-firewall-bouncer    v0.0.22-debian-pragmatic-f64e94b59a948717c3dc848f9abebb27b5974714
 proxmox-iptables  10.X.X.X    ✔️      2022-02-18T22:41:11Z  crowdsec-firewall-bouncer    v0.0.22-debian-pragmatic-f64e94b59a948717c3dc848f9abebb27b5974714
 npm-openresty     172.18.0.3  ✔️      2022-02-18T22:41:07Z  crowdsec-openresty-bouncer   v0.0.1
-----------------------------------------------------------------------------------------------------------------------------------------------------------
❯ cscli bouncers list
-----------------------------------------------------------------------------------------------------------------------------------------------------------
 NAME              IP ADDRESS  VALID  LAST API PULL         TYPE                         VERSION
-----------------------------------------------------------------------------------------------------------------------------------------------------------
 ZM-IPtables       10.X.X.X    ✔️      2022-02-18T22:41:14Z  crowdsec-firewall-bouncer    v0.0.22-debian-pragmatic-f64e94b59a948717c3dc848f9abebb27b5974714
 Local-cloudflare  ::1         ✔️      2022-02-18T22:41:14Z  crowdsec-cloudflare-bouncer  v0.0.8-debian-pragmatic-38768ad6e47bc7ce058668ef286de303f897d705
 Local-IPtables    ::1         ✔️      2022-02-18T22:41:16Z  crowdsec-firewall-bouncer    v0.0.22-debian-pragmatic-f64e94b59a948717c3dc848f9abebb27b5974714
 proxmox-iptables  10.X.X.X    ✔️      2022-02-18T22:41:11Z  crowdsec-firewall-bouncer    v0.0.22-debian-pragmatic-f64e94b59a948717c3dc848f9abebb27b5974714
 npm-openresty     172.18.0.3  ✔️      2022-02-18T22:41:17Z  crowdsec-openresty-bouncer   v0.0.1
-----------------------------------------------------------------------------------------------------------------------------------------------------------
commented

Example docker-compose.yaml, literally just change the image: and add the new ENV vars.

version: "3"
services:
  npm:
    #image: 'jc21/nginx-proxy-manager:latest'
    image: 'baudneo/nginx-proxy-manager:latest'
    restart: always
    container_name: npm-crowdsec
    ports:
      # Public HTTP Port:
      - '80:80'
      # Public HTTPS Port:
      - '443:443'
      # Admin Web Port:
      - '81:81'
    environment:
      ADMIN_PANEL_LOG: "1"
      CROWDSEC_BOUNCER: "1"
      OPENRESTY_DEBUG: "0"

      CROWDSEC_LAPI: "http://IP TO CROWDSEC LOCAL API:8080"
      CROWDSEC_KEY: "xxxxxxxxxxxxxxxxxxxxxxxx"

      # These are the settings to access your db
      DB_MYSQL_HOST: "db"
      DB_MYSQL_PORT: 3306
      DB_MYSQL_USER: "npm"
      DB_MYSQL_PASSWORD: "PASSw0rD"
      DB_MYSQL_NAME: "npm"
      # If you would rather use Sqlite uncomment this
      # and remove all DB_MYSQL_* lines above
      # DB_SQLITE_FILE: "/data/database.sqlite"
      # Uncomment this if IPv6 is not enabled on your host
      # DISABLE_IPV6: 'true'
    volumes:
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt
    depends_on:
      - db
  db:
    image: 'jc21/mariadb-aria:latest'
    restart: always
    container_name: npm_db
    environment:
      MYSQL_ROOT_PASSWORD: 'xxXXxxXXXxxxXXX'
      MYSQL_DATABASE: 'npm'
      MYSQL_USER: 'npm'
      MYSQL_PASSWORD: 'DB Passw0rd'
    volumes:
      - ./data/mysql:/var/lib/mysql
commented

@baudneo - This is excellent. Thank you for everyone's contributions. I will be testing this in my lab soonish.

commented

I have added ModSecurity to a new image with CrowdSec

Now CrowdSec and ModSecurity are installed and working. The new image that has modsec is docker pull baudneo/nginx-proxy-manager:cs-modsec. I installed OWASP-CRS for modsec. The modsec audit log is in /data/logs/modsec_audit.log by default, you can turn it off by editing /data/modsec/modsecurity.conf.

NOTE

  • All config for modsec is in /data/modsec which is symbolically linked to /etc/nginx/modsec
  • The OWASP rules are in /data/modsec/ruleset/
  • The main rules file is /data/modsec/main.conf which just Includes the OWASP ruleset.
  • The only ENV VAR for modsec is MODSEC_CREATE which will force create the directory structure into /data/modsec. This is usually only run on the first run of the modsec image when this directory does not exist. this ENV VAR will allow you to force it for whatever reason.

To enable modsec you need to add these 2 directives to the config somewhere.

modsecurity on;
modsecurity_rules_file /etc/nginx/modsec/main.conf;

Here are some pointers.

  • To enable modsec for ALL HTTP hosts put the above directives in /data/nginx/custom/http_top.conf.
  • To enable modsec for ALL STREAM hosts put the above directives in /data/nginx/custom/stream.conf.
  • To enable modsec for ONE certain HTTP host put the above directives in the 'Advanced' tab configuration.
  • To enable modsec for ONE certain HTTP host in a certain LOCATION put the above directives in the location {} block instead of outside of it in the 'Advanced' tab config.
  • To enable on a per-stream basis you need to enable the streams in the dashboard and then edit the stream file by hand to add those directives into the stream {} block you want.

Some logs from modsec_audit.log to confirm modsecurity is working. This was a test using ?exec=/bin/bash in the URI.

---C3KCgGTx---F--
HTTP/2.0 403
Server: nginx
Date: Tue, 22 Feb 2022 06:16:02 GMT
Content-Length: 552
Content-Type: text/html
Connection: close
Strict-Transport-Security: max-age=63072000;includeSubDomains; preload

---C3KCgGTx---H--
ModSecurity: Warning. Matched "Operator `PmFromFile' with parameter `unix-shell.data' against variable `ARGS:exec' (Value: `/bin/bash' ) [file "/etc/nginx/modsec/ruleset/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf"] [line "480"] [id "932160"] [rev ""] [msg "Remote Command Execution: Unix Shell Code Found"] [data "Matched Data: bin/bash found within ARGS:exec: /bin/bash"] [severity "2"] [ver "OWASP_CRS/3.3.2"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-shell"] [tag "platform-unix"] [tag "attack-rce"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/152/248/88"] [tag "PCI/6.5.2"] [hostname "172.18.0.4"] [uri "/zm/"] [unique_id "1645510562"] [ref "o1,8v14,9t:urlDecodeUni,t:cmdLine,t:normalizePath,t:lowercase"]
ModSecurity: Access denied with code 403 (phase 2). Matched "Operator `Ge' with parameter `5' against variable `TX:ANOMALY_SCORE' (Value: `5' ) [file "/etc/nginx/modsec/ruleset/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "80"] [id "949110"] [rev ""] [msg "Inbound Anomaly Score Exceeded (Total Score: 5)"] [data ""] [severity "2"] [ver "OWASP_CRS/3.3.2"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [hostname "172.18.0.4"] [uri "/zm/"] [unique_id "1645510562"] [ref ""]

I don't know JavaScript so I am unable to add to the UI to make things pretty. Ideally there would be a ModSecurity tab that has a toggle switch for on/off and a text input box for the rules file. This tab would be in the proxyhost modal where you set what domain/target host/target port etc. It would be nice to have some sort of UI for corwdsec as well to turn it off and on and to edit the config file. I don't think having a rule editor for modsec would matter much, but it would be nice to have it.

I might take the time to teach myself some JS just to implement this to make it all pretty. Things are working for CrowdSec and ModSecurity though. The people who want that functionality should not have an issue with configuring things from the CLI anyways, for now.

commented

Will this also work with the CrowdSec agent installed directly on Docker host?

commented

Will this also work with the CrowdSec agent installed directly on Docker host?

Yes just edit the bouncer config file and put the ip:port of the docker host in the API_URL. Do not use "localhost:8080" use the actual IP for your docker host. If you have any issues when starting the container issue docker logs --follow <container name> to see what the logs are saying. I also recommend tail /path/to/data/logs/*_error.log to see if any Lua or crowdsec errors are in those logs.

If you mean using docker networks I am not sure as I do not have my system setup that way. I am sure there is a way to make that work.

I am currently building the multiarch image base images and will be pushing multi arch images soon.

commented

Thanks. I am using a Docker bridged network totally standardized. If one configures the agent to listen on 0.0.0.0 it can be reached from the container using it's docker network ip. Remember to firewall the port so it's not internet exposed (which reminds me that I asked the devs if there's a CrowdSec parser + scenario for CrowdSec. There should be. Very meta :-)

commented

I just ugrade my NPM with "baudneo/nginx-proxy-manager:cs-modsec" on my OpenWrt server (with NextCloud).
so all broken now :

stderr: standard_init_linux.go:228: exec user process caused: exec format error

# uname -ar
Linux STARGATE 5.4.124 #0 SMP Sun Jun 13 22:02:19 2021 aarch64 GNU/Linux

Back to the 'jc21/nginx-proxy-manager:latest'
okay again...

My own use for Nginx-Proxy-Manager is to check the logs externally from the dockerized side !
just with:

    volumes:
      - /srv/NGINX/data/nginx-proxy-manager:/data
      - /srv/NGINX/letsencrypt:/etc/letsencrypt
      - /srv/NGINX/logs:/var/log/nginx

ad in the main OpenWrt host :

$ cat /etc/crowdsec/acquis.d/npm.yaml 

# NginxProxyManager
filenames:
#  - ~/data/logs/*.log
# Dockerized NGINX PROXY MANAGER
  - /srv/NGINX/data/nginx-proxy-manager/logs/*.log
labels:
  type: nginx-proxy-manager

with intalled collection : crowdsecurity/nginx-proxy-manager

commented

@erdoukki - What's the CPU architecture for your OpenWRT server?

commented

architecture : aarch64
cortex-a53 arm64
Marvell armada37xx

commented

I don't see an ARM build for baudneo/nginx_proxy_manager on dockerhub . Which is most likely causing your issue. You could try editing the Dockerfile to build an ARM image or ask the owner to include ARM architecture.

commented

Good idea.
Thanks.

commented

#1875 (comment)

This is an automated message from CI:

Docker Image for build 5 is available on DockerHub as jc21/nginx-proxy-manager:github-pr-1875

Note: ensure you backup your NPM instance before testing this PR image! Especially if this PR contains database changes.

I am trying with this "PR"... ;-)

commented

Working !
But the dockerhub image from the PR do not include the modifications for CrowdSec integration.
Bad attempt.
Badly the actual PR do not have any checks from CI which to take dockerhub for tests on arm64 !
@baudneo Why you're modifications proposal do not include arm64 ?
I check quickly your code but I still don't find the reason.

commented

@erdoukki I have multi-arch built now. ModSec is installed and loaded by default. User needs to configure it to start detecting though. See the PR for config. #1867

docker pull baudneo/nginx-proxy-manager:latest

commented

Thanks!

Just tested;

root@STARGATE:~# cscli bouncers add docker-crowdsec-npm-bouncer
Api key for 'docker-crowdsec-npm-bouncer':

   xxxx

Please keep this key since you will not be able to retrieve it!

/srv/NGINX/data/nginx-proxy-manager/crowdsec/crowdsec-openresty-bouncer.conf

...
API_KEY=xxxx
...

docker-compose down
docker-compose up -d

root@STARGATE:/srv/NGINX# docker-compose restart
Restarting npm-crowdsec ... done
root@STARGATE:/srv/NGINX# cscli bouncers list
---------------------------------------------------------------------------------------------------------------------------
 NAME                         IP ADDRESS  VALID  LAST API PULL         TYPE                       VERSION                  
---------------------------------------------------------------------------------------------------------------------------
 crowdsec-firewall-bouncer    ::1         ✔️      2022-03-02T09:42:38Z  crowdsec-firewall-bouncer  v0.0.21-openwrt-0.0.21-3 
 docker-crowdsec-npm-bouncer              ✔️      2022-03-02T09:53:29Z                                                      
---------------------------------------------------------------------------------------------------------------------------

but still issue :

time="02-03-2022 10:55:57" level=error msg="auth api key error: select bouncer: ent: bouncer not found: unable to query"
commented

Okay, just rebuild the docker and now works fine:

INFO[02-03-2022 01:56:04 PM] Local Api Bouncers Metrics:                  
+-----------------------------+----------------------+--------+------+
|           BOUNCER           |        ROUTE         | METHOD | HITS |
+-----------------------------+----------------------+--------+------+
| crowdsec-firewall-bouncer   | /v1/decisions/stream | GET    | 1067 |
| docker-crowdsec-npm-bouncer | /v1/decisions/stream | GET    |   37 |
+-----------------------------+----------------------+--------+------+
commented

I have added ModSecurity to a new image with CrowdSec

@baudneo Where can I submit issues for your fork? I have two:

  1. Specifying enviromental variables like this doesn't work. The bouncer is not started unless I edit the crowdsec-openresty-bouncer.conf file.
    environment:
      CROWDSEC_BOUNCER: "1"
      CROWDSEC_LAPI: "http://1xxxx:8080"
      CROWDSEC_KEY: "xxxx"
  1. There's a memory leak in the current openresty bouncer that currently is fixed with a workaround. Will you implement it in your fork?

Lastly I have a suggestion: Could you implement more environmental variables so everything in the bouncer configuration can configured like this. Especially the reCAPTCHA stuff..

Thanks - keep up the good work!

commented

I have added ModSecurity to a new image with CrowdSec

Now CrowdSec and ModSecurity are installed and working. The new image that has modsec is docker pull baudneo/nginx-proxy-manager:cs-modsec. I installed OWASP-CRS for modsec. The modsec audit log is in /data/logs/modsec_audit.log by default, you can turn it off by editing /data/modsec/modsecurity.conf.

NOTE

  • All config for modsec is in /data/modsec which is symbolically linked to /etc/nginx/modsec
  • The OWASP rules are in /data/modsec/ruleset/
  • The main rules file is /data/modsec/main.conf which just Includes the OWASP ruleset.
  • The only ENV VAR for modsec is MODSEC_CREATE which will force create the directory structure into /data/modsec. This is usually only run on the first run of the modsec image when this directory does not exist. this ENV VAR will allow you to force it for whatever reason.

To enable modsec you need to add these 2 directives to the config somewhere.

modsecurity on;
modsecurity_rules_file /etc/nginx/modsec/main.conf;

Here are some pointers.

  • To enable modsec for ALL HTTP hosts put the above directives in /data/nginx/custom/http_top.conf.
  • To enable modsec for ALL STREAM hosts put the above directives in /data/nginx/custom/stream.conf.
  • To enable modsec for ONE certain HTTP host put the above directives in the 'Advanced' tab configuration.
  • To enable modsec for ONE certain HTTP host in a certain LOCATION put the above directives in the location {} block instead of outside of it in the 'Advanced' tab config.
  • To enable on a per-stream basis you need to enable the streams in the dashboard and then edit the stream file by hand to add those directives into the stream {} block you want.

Some logs from modsec_audit.log to confirm modsecurity is working. This was a test using ?exec=/bin/bash in the URI.

---C3KCgGTx---F--
HTTP/2.0 403
Server: nginx
Date: Tue, 22 Feb 2022 06:16:02 GMT
Content-Length: 552
Content-Type: text/html
Connection: close
Strict-Transport-Security: max-age=63072000;includeSubDomains; preload

---C3KCgGTx---H--
ModSecurity: Warning. Matched "Operator `PmFromFile' with parameter `unix-shell.data' against variable `ARGS:exec' (Value: `/bin/bash' ) [file "/etc/nginx/modsec/ruleset/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf"] [line "480"] [id "932160"] [rev ""] [msg "Remote Command Execution: Unix Shell Code Found"] [data "Matched Data: bin/bash found within ARGS:exec: /bin/bash"] [severity "2"] [ver "OWASP_CRS/3.3.2"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-shell"] [tag "platform-unix"] [tag "attack-rce"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/152/248/88"] [tag "PCI/6.5.2"] [hostname "172.18.0.4"] [uri "/zm/"] [unique_id "1645510562"] [ref "o1,8v14,9t:urlDecodeUni,t:cmdLine,t:normalizePath,t:lowercase"]
ModSecurity: Access denied with code 403 (phase 2). Matched "Operator `Ge' with parameter `5' against variable `TX:ANOMALY_SCORE' (Value: `5' ) [file "/etc/nginx/modsec/ruleset/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "80"] [id "949110"] [rev ""] [msg "Inbound Anomaly Score Exceeded (Total Score: 5)"] [data ""] [severity "2"] [ver "OWASP_CRS/3.3.2"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [hostname "172.18.0.4"] [uri "/zm/"] [unique_id "1645510562"] [ref ""]

I don't know JavaScript so I am unable to add to the UI to make things pretty. Ideally there would be a ModSecurity tab that has a toggle switch for on/off and a text input box for the rules file. This tab would be in the proxyhost modal where you set what domain/target host/target port etc. It would be nice to have some sort of UI for corwdsec as well to turn it off and on and to edit the config file. I don't think having a rule editor for modsec would matter much, but it would be nice to have it.

I might take the time to teach myself some JS just to implement this to make it all pretty. Things are working for CrowdSec and ModSecurity though. The people who want that functionality should not have an issue with configuring things from the CLI anyways, for now.

Do I still have to add the log path to /etc/crowdsec/acquis.yaml here?

commented

Awesome! Thanks for updating it!

commented

Hello baudneo, thanks for this new image. I've tried to update, however docker can't pull the image, it complains about missing manifest. I've verified docker hub and think the image is not updated, is that ok?

$ docker pull baudneo/nginx-proxy-manager:cs-modsec Error response from daemon: manifest for baudneo/nginx-proxy-manager:cs-modsec not found: manifest unknown: manifest unknown

commented

Any chance this is being merged back into the base jc21/nginx-proxy-manager image ?

commented

ModSec is EoL now/soon and also has known memory leaks which are HUGE, so I won't be building images with it. I am currently working on ZoneMinder ML object detection stuff, as soon as I am done that I will create a new image with current NPM and add crowdsec openresty bouncer into it.

I think the options for WAF are slim, coraza and open-appsec which I may or may not look into adding either of them into an NPM image.

Coraza parses modsec security language, so the rules files like OWASP (who develop coraza as well) lists can be used with it. Furthermore, Coraza seems to need to be implemented in a Go language app. It is not as simple as download, point it at configs/rules and run it, someone needs to write a Go app using coraza libraries. That may have changed as I have not kept up with Coraza.

open-appsec uses ML instead of signatures for threat detection which is neat but, I can't speak to its effectiveness. At the time when I realized ModSec had issues, open-appsec did not have a nginx connector, which I think they do now. They require you to have a cloud control panel of sorts, so IDK how feasible it is to integrate into NPM if you can't do a stand alone install. Again, things may have changed as I haven't kept up on open-appsec either.

I stopped using NPM in favour of Cloudflare tunnels but, CF's new browser security check stuff is causing me grief so, I will need to move back to NPM.

commented

Is there any hope of this being revived? I would love to have the option of integrating Crowdsec with NPM.

commented

@baudneo

I have the following problem. When I want to login to npm and change the email as it asks me, the response is Forbidden.

If I want to create a new user, I can't apply rights or change the password = Forbidden.
I saw this problem in a very old version of npm without having been confronted with the problem.
But I read it on the web.

I have a working npm instance which I temporarily shut down, Crowdesc installed with no errors.
The changes made in the conf files are ok, I created a separate base for this fork baudneo/nginx-proxy-manager:latest in mariadb.

My concern is only on the change of email and password.
So I tell myself that further on it might not be so good as well.

Note : The correct permissions are applied to the folders in your fork. (I'm under unraid on Dell PowerEdge)

commented

Issue is now considered stale. If you want to keep it open, please comment 👍

commented

up news ?