nginx: [warn] duplicate network
harryqt opened this issue · comments
root@ex44-2261978 ~ # nginx -t
nginx: [warn] duplicate network "138.199.57.151", value: "0", old value: "1" in /etc/nginx/conf.d/globalblacklist.conf:18890
nginx: [warn] duplicate network "143.244.38.129", value: "0", old value: "1" in /etc/nginx/conf.d/globalblacklist.conf:18906
nginx: [warn] duplicate network "195.181.163.194", value: "0", old value: "1" in /etc/nginx/conf.d/globalblacklist.conf:19001
nginx: [warn] duplicate network "5.188.120.15", value: "0", old value: "1" in /etc/nginx/conf.d/globalblacklist.conf:19128
nginx: [warn] duplicate network "89.187.173.66", value: "0", old value: "1" in /etc/nginx/conf.d/globalblacklist.conf:19175
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
It's not a bug nor can it be fixed, this is desired behaviour of the blocker. Daily updates of IP blacklists cause some well known IP's and ranges to be blacklisted old value "1" these are then whitelisted at the very end of globalblocklist which is the order of loading which then sets IP's we know are good to their new value "0" thereby whitelisting them. It has been this way since day 1 of the blocker and will remain this way. These are simple [WARN] messages not [EMERG] messages and they do not affect the operation of Nginx in any way whatsoever.
Thank you for clearing that up, I appreciate it. 👍
I wrote a Python script to remove the exact duplicate IP addresses/ranges that are listed more than one time, leaving only the last occurrence. I injected the invocation of my script into /usr/local/sbin/update-ngxblocker script, right after the script moves the new globalblacklist.conf into it's designated place. All seems to work as expected: no more flooding of my nginx logs with duplicate network warnings. My question is that is there some hidden reason why the IPs are first "1" and then later overwritten with "0" I should worry about?
Here's my script: https://github.com/henno/nginx-ultimate-bad-bot-blocker-duplicate-ip-remover
@henno Thank you so much for the script.
You're welcome. I actually haven't written any Python before. I just chatgptd my way through it. :D
Thanks for the warning, will wait a month before deploying to production. 😂 bc I don’t trust ChatGPT
Thanks for the warning, will wait a month before deploying to production. 😂 bc I don’t trust ChatGPT
That's a wise approach, though I wouldn't be overly concerned. I've been developing in a variety of programming languages since 1993, just not in Python. It took me roughly 2 hours and about 20-30 iterations with ChatGPT to refine the code to a point where it was completely free of errors. To ensure its accuracy, I conducted tests by making commits before and after adjustments, then examining the differences using the Jetbrains IDE diff viewer. Initially, there were numerous issues, including significant sections of the file being moved around and only the IP part of CIDR notation considered for duplicates (the regex didn't include the subnet) and IPv6 being omitted from search. However, after some adjustments, the only changes made were necessary deletions. The process effectively removes all duplicate instances, retaining only the last occurrence of the IP/range.