New "super rate limiting" zone (1r/m) in addition of the current one (2r/s)
bigrck64 opened this issue Β· comments
Hello Mitchell
Since a week, ClaudeBot (UA) is harvesting my server from a lot of public IP.
I do not want to completly block it, but I want to heavily rate limited it.
Problem: current limiter setting of 2r/s is not enough restrictive.
(because this bot is using 100+ different IP, I have 2r/s allowed for each IP -> this is too high for my server)
Current limiter:
map $bad_bot $bot_iplimit {
0 "";
1 "";
2 $binary_remote_addr;
}
# Rate limiting will only take effect if on any User-Agents with a value of 2
limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m;
limit_req_zone $bot_iplimit zone=bot2_reqlimitip:16m rate=2r/s;
Would you be able to introduce a second zone with a more restrictive limit like 1r/m ?
It would be perfect if I could use another value to activate this new settings for UA (like 4 ?).
# Super rate limiting will only take effect if on any User-Agents with a value of 4
limit_conn_zone $bot_iplimit zone=bot3_connlimit:16m;
limit_req_zone $bot_iplimit zone=bot3_reqlimitip:16m rate=1r/m;
Moreover, this new setting could be helpfull for other user like i300320 : #184
Thanks for considering this request π
And a big thumb up for this project !
Hi @bigrck64
I'm a bit tied down with some other work. Can you possibly do a test, by manually modifying your globalblacklist.conf as follows
# 1. MAP BAD BOTS TO OUR RATE LIMITER FUNCTION
# --------------------------------------------
map $bad_bot $bot_iplimit {
0 "";
1 "";
2 $binary_remote_addr;
4 $binary_remote_addr;
}
# --------------------------
# 2. SET RATE LIMITING ZONES
# --------------------------
# BAD BOT RATE LIMITING ZONE
# Rate limiting will only take effect if on any User-Agents with a value of 2
limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m;
limit_req_zone $bot_iplimit zone=bot2_reqlimitip:16m rate=2r/s;
# BAD BOT SUPER RATE LIMITING ZONE
# Super Rate limiting will only take effect if on any User-Agents with a value of 4
limit_conn_zone $bot_iplimit zone=bot4_connlimit:16m;
limit_req_zone $bot_iplimit zone=bot4_reqlimitip:16m rate=1r/s;
Let me know if this works
Hi Mitchell
Thanks for quick reply !
It seems to work so far :)
Please just change the 1r/s to 1r/m π
limit_req_zone $bot_iplimit zone=bot4_reqlimitip:16m rate=1r/m;
Thanks !
Hi Mitchell
Thanks for quick reply ! It seems to work so far :)
Please just change the 1r/s to 1r/m π
limit_req_zone $bot_iplimit zone=bot4_reqlimitip:16m rate=1r/m;
Thanks !
@bigrck64
Done, feature added - docs updated and tests added into build tests.
Awesome ! π
Maybe you should change
### 1 = allowed or rate limited less restrictive
### 2 = rate limited more
### 3 = block completely
### 4 = super rate limited
to
### 1 = allowed
### 2 = rate limited
### 3 = block completely
### 4 = super rate limited
Because for me, 1 does not perform any rate limited task, no ?
@bigrck64 correct 1
essentially does nothing, same as 0
bit too late to change that.
Will update docs.
Maybe you should change
### 1 = allowed or rate limited less restrictive ### 2 = rate limited more ### 3 = block completely ### 4 = super rate limited
to
### 1 = allowed ### 2 = rate limited ### 3 = block completely ### 4 = super rate limited
Because for me, 1 does not perform any rate limited task, no ?
Have done - but explained what 1 actually is/was for and may still be implemented if anyone wants an even less agressive rate limiting zone.
Hum, I download your latest version and I feel that "Claudebot" is going into the rate=2r/s instead of rate=1r/m zone :(
Maybe with 2 or 4 it just send the binary_remote_addr into $bot_iplimit:
map $bad_bot $bot_iplimit {
0 "";
1 "";
2 $binary_remote_addr;
4 $binary_remote_addr;
}
Which just finish into
# Rate limiting will only take effect if on any User-Agents with a value of 2
limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m;
limit_req_zone $bot_iplimit zone=bot2_reqlimitip:16m rate=2r/s;
So the solution may be something like
limit_conn_zone $bot_iplimit2 zone=bot4_connlimit:16m;
limit_req_zone $bot_iplimit2 zone=bot4_reqlimitip:16m rate=1r/m;
and
map $bad_bot $bot_iplimit {
0 "";
1 "";
2 $binary_remote_addr;
4 "";
}
map $bad_bot $bot_iplimit2 {
0 "";
1 "";
2 "";
4 $binary_remote_addr;
}
?
Please add this manually to your local globalblacklist.conf and see if nginx throws any errors - I'm a bit tied down with other stuff
# ============================================
# BEGIN SECTION 4 - ACTIVATE BLOCKER FUNCTIONS
# ============================================
# --------------------------------------------
# 4.1. MAP BAD BOTS TO OUR RATE LIMITER FUNCTION
# --------------------------------------------
map $bad_bot $bot_iplimit {
0 "";
1 "";
2 $binary_remote_addr;
4 "";
}
# --------------------------------------------------
# 4.2. MAP BAD BOTS TO OUR SUPER RATE LIMITER FUNCTION
# --------------------------------------------------
map $bad_bot $bot_iplimit2 {
0 "";
1 "";
2 "";
4 $binary_remote_addr;
}
# --------------------------
# 4.3. SET RATE LIMITING ZONES
# --------------------------
# BAD BOT RATE LIMITING ZONE
# Rate limiting will only take effect if on any User-Agents with a value of 2
limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m;
limit_req_zone $bot_iplimit zone=bot2_reqlimitip:16m rate=2r/s;
# BAD BOT SUPER RATE LIMITING ZONE
# Super Rate limiting will only take effect if on any User-Agents with a value of 4
limit_conn_zone $bot_iplimit2 zone=bot4_connlimit:16m;
limit_req_zone $bot_iplimit2 zone=bot4_reqlimitip:16m rate=1r/m;
Thanks !
No problem with the new nginx config :)
The rate limiting seems fine. (edit: it is not π’)
I can not run your test-blocker-rate-limiting.sh on my prod server, but maybe you will be able to confirm the super rate limiting is effective with it ?
Hello Mitchell
I worked on the problem π
So I setup a new server to be able to understand your "test-blocker-rate-limiting.sh"
And I improved it with the following change:
- Add the duration and change from 80 to 60 curl requests with the response code:
truncate -s 0 ${ratelimittestfile}
SECONDS=0
curl -s -m 90 -A "GoogleBot" -w "%{http_code}\n" -o /dev/null http://localhost:80 >> ${ratelimittestfile} &
- Wait for all the curl to finish, then display the duration and the number of response code 200 or 503
curl -s -m 90 -A "GoogleBot" -w "%{http_code}\n" -o /dev/null http://localhost:80 >> ${ratelimittestfile} &
curl -s -m 90 -A "GoogleBot" -w "%{http_code}\n" -o /dev/null http://localhost:80 >> ${ratelimittestfile}
echo "${bold}${yellow}------------------------------"
echo "${bold}${yellow}Wait for all 60 curl to finish"
echo "${bold}${yellow}------------------------------"
printf "\n"
wait
echo "Duration = $SECONDS seconds"
nb200=`cat ${ratelimittestfile} |grep 200 |wc -l`
echo "Count the number of 200 (response code) = $nb200";
nb503=`cat ${ratelimittestfile} |grep 503 |wc -l`
echo "Count the number of 503 (response code) = $nb503";
printf "\n"
if [ $SECONDS -lt 30 ]
then
echo "${bold}${green}PASSED - ${green}GoogleBot was ${bold}${green}RATE LIMITED"
else
echo "${bold}${red}FAILED - ${red}GoogleBot was ${bold}${red}NOT RATE LIMITED"
fi
printf "\n"
}
- Same things for Applebot
truncate -s 0 ${ratelimittestfile}
SECONDS=0
curl -s -m 90 -A "Applebot" -w "%{http_code}\n" -o /dev/null http://localhost:80 >> ${ratelimittestfile} &
And
curl -s -m 90 -A "Applebot" -w "%{http_code}\n" -o /dev/null http://localhost:80 >> ${ratelimittestfile} &
curl -s -m 90 -A "Applebot" -w "%{http_code}\n" -o /dev/null http://localhost:80 >> ${ratelimittestfile}
echo "${bold}${yellow}------------------------------"
echo "${bold}${yellow}Wait for all 60 curl to finish"
echo "${bold}${yellow}------------------------------"
printf "\n"
wait
echo "Duration = $SECONDS seconds"
nb200=`cat ${ratelimittestfile} |grep 200 |wc -l`
echo "Count the number of 200 (response code) = $nb200";
nb503=`cat ${ratelimittestfile} |grep 503 |wc -l`
echo "Count the number of 503 (response code) = $nb503";
printf "\n"
if [ $SECONDS -gt 30 ]
then
echo "${bold}${green}PASSED - ${green}Applebot was ${bold}${green}SUPER RATE LIMITED"
else
echo "${bold}${red}FAILED - ${red}Applebot was ${bold}${red}NOT SUPER RATE LIMITED"
fi
printf "\n"
Conclusion
This new version of test-blocker-rate-limiting.sh show us the duration time of all curl and the % of request ok/rejected π
It also confirm that the current implementation of "super rate limited" have the same result than "rate limited".
TEST WITH 2r/s =
limit_req_zone $bot_iplimit zone=bot2_reqlimitip:16m rate=2r/s;
limit_req_zone $bot_iplimit zone=bot4_reqlimitip:16m rate=2r/s;
RESULT
------------------------------
Wait for all 60 curl to finish
------------------------------
Duration = 5 seconds
Count the number of 200 (response code) = 11
Count the number of 503 (response code) = 49
PASSED - GoogleBot was RATE LIMITED
------------------------------
Wait for all 60 curl to finish
------------------------------
Duration = 5 seconds
Count the number of 200 (response code) = 10
Count the number of 503 (response code) = 50
FAILED - Applebot was NOT SUPER RATE LIMITED
TEST with 1r/m
limit_req_zone $bot_iplimit zone=bot2_reqlimitip:16m rate=1r/m;
limit_req_zone $bot_iplimit zone=bot4_reqlimitip:16m rate=1r/m;
RESULT
------------------------------
Wait for all 60 curl to finish
------------------------------
Duration = 90 seconds
Count the number of 200 (response code) = 0
Count the number of 503 (response code) = 58
PASSED - GoogleBot was RATE LIMITED
------------------------------
Wait for all 60 curl to finish
------------------------------
Duration = 90 seconds
Count the number of 200 (response code) = 0
Count the number of 503 (response code) = 58
PASSED - Applebot was SUPER RATE LIMITED
edit1: make the curl wait 90 sec max to not freeze completly the script with nginx super rate limited (1r/m)
edit2: add the total duration of 60 curl tests (BASH native)
edit3: slow down from 80 to 60 curl request to not slow the test at 2 minuts with normal "rate limited" settings (2r/s)
edit4: change [if] from number of 503 to if duration of replied (more stable)
Latest test-blocker-rate-limiting.sh (v2) :
test-blocker-rate-limiting.zip
I will now work into nginx globalblacklist.conf to see if I can find a way to make the "super rate limited" to work along with "rate limited" π
Hi Mitchell,
So with the updated test-blocker-rate-limiting.sh, I was able to test and find a solution for nginx and "super rate limiter" π
Your modification to globalblacklist.conf was perfect, so we keep that:
# ============================================
# BEGIN SECTION 4 - ACTIVATE BLOCKER FUNCTIONS
# ============================================
# --------------------------------------------
# 4.1. MAP BAD BOTS TO OUR RATE LIMITER FUNCTION
# --------------------------------------------
map $bad_bot $bot_iplimit {
0 "";
1 "";
2 $binary_remote_addr;
4 "";
}
# --------------------------------------------------
# 4.2. MAP BAD BOTS TO OUR SUPER RATE LIMITER FUNCTION
# --------------------------------------------------
map $bad_bot $bot_iplimit2 {
0 "";
1 "";
2 "";
4 $binary_remote_addr;
}
# --------------------------
# 4.3. SET RATE LIMITING ZONES
# --------------------------
# BAD BOT RATE LIMITING ZONE
# Rate limiting will only take effect if on any User-Agents with a value of 2
limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m;
limit_req_zone $bot_iplimit zone=bot2_reqlimitip:16m rate=2r/s;
# BAD BOT SUPER RATE LIMITING ZONE
# Super Rate limiting will only take effect if on any User-Agents with a value of 4
limit_conn_zone $bot_iplimit2 zone=bot4_connlimit:16m;
limit_req_zone $bot_iplimit2 zone=bot4_reqlimitip:16m rate=1r/m;
We only forgot the bot_iplimit2 declaration into blockbots.conf !
limit_conn bot2_connlimit 10;
limit_req zone=bot2_reqlimitip burst=10;
limit_conn bot4_connlimit 10;
limit_req zone=bot4_reqlimitip burst=10;
if ($bad_bot = '3') {
return 444;
This lead to a working validation with new test-blocker-rate-limiting.sh:
- Googlebot (2r/s) got responses in less than 30 sec (2 sec)
- Applebot (1r/m) got responses in more than 30 sec (90 sec)
------------------------------
Wait for all 60 curl to finish
------------------------------
Duration = 5 seconds
Count the number of 200 (response code) = 11
Count the number of 503 (response code) = 49
PASSED - GoogleBot was RATE LIMITED
------------------------------
Wait for all 60 curl to finish
------------------------------
Duration = 90 seconds
Count the number of 200 (response code) = 0
Count the number of 503 (response code) = 58
PASSED - Applebot was SUPER RATE LIMITED
Feel free to update test-blocker-rate-limiting.sh, and please patch globalblacklist.conf + blockbots.conf π
Hi Mitchell,
So with the updated test-blocker-rate-limiting.sh, I was able to test and find a solution for nginx and "super rate limiter" π
Your modification to globalblacklist.conf was perfect, so we keep that:
# ============================================ # BEGIN SECTION 4 - ACTIVATE BLOCKER FUNCTIONS # ============================================ # -------------------------------------------- # 4.1. MAP BAD BOTS TO OUR RATE LIMITER FUNCTION # -------------------------------------------- map $bad_bot $bot_iplimit { 0 ""; 1 ""; 2 $binary_remote_addr; 4 ""; } # -------------------------------------------------- # 4.2. MAP BAD BOTS TO OUR SUPER RATE LIMITER FUNCTION # -------------------------------------------------- map $bad_bot $bot_iplimit2 { 0 ""; 1 ""; 2 ""; 4 $binary_remote_addr; } # -------------------------- # 4.3. SET RATE LIMITING ZONES # -------------------------- # BAD BOT RATE LIMITING ZONE # Rate limiting will only take effect if on any User-Agents with a value of 2 limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m; limit_req_zone $bot_iplimit zone=bot2_reqlimitip:16m rate=2r/s; # BAD BOT SUPER RATE LIMITING ZONE # Super Rate limiting will only take effect if on any User-Agents with a value of 4 limit_conn_zone $bot_iplimit2 zone=bot4_connlimit:16m; limit_req_zone $bot_iplimit2 zone=bot4_reqlimitip:16m rate=1r/m;
We only forgot the bot_iplimit2 declaration into blockbots.conf !
limit_conn bot2_connlimit 10; limit_req zone=bot2_reqlimitip burst=10; limit_conn bot4_connlimit 10; limit_req zone=bot4_reqlimitip burst=10; if ($bad_bot = '3') { return 444;
This lead to a working validation with new test-blocker-rate-limiting.sh:
- Googlebot (2r/s) got responses in less than 30 sec (2 sec)
- Applebot (1r/m) got responses in more than 30 sec (90 sec)
------------------------------ Wait for all 60 curl to finish ------------------------------ Duration = 5 seconds Count the number of 200 (response code) = 11 Count the number of 503 (response code) = 49 PASSED - GoogleBot was RATE LIMITED ------------------------------ Wait for all 60 curl to finish ------------------------------ Duration = 90 seconds Count the number of 200 (response code) = 0 Count the number of 503 (response code) = 58 PASSED - Applebot was SUPER RATE LIMITED
Feel free to update test-blocker-rate-limiting.sh, and please patch globalblacklist.conf + blockbots.conf π
Done, please grab latest of both files and confirm.
Tested on my server and working OK - I still need to reintroduce the BETA tests in this repo which broke some time ago.
By default I have left the new super rate limiting zone commented out in blockbots.conf users can decide to enable themselves. It threw an error during the build so I need to as i said reintroduce the beta tests again which test stuff like this without causing any potential breaking changes.
Hi Mitchell,
Thanks for the patch to the master branch !
I uninstall / reinstall UBBB to my test and production server (Linux Debian), and it's working as expected, nice ! π
I have no error with the uncommented following lines in blockbots.conf, but it's fine like this, user can decide themself to active the "super rate limiter" zone or not π
# Uncomment below lines for super rate limiting feature
limit_conn bot4_connlimit 10;
limit_req zone=bot4_reqlimitip burst=10;
Thanks for all !