Server can go down if "Check users for spam" left blank and over 100+ listed

[Mark Rejhon]

I had a situation where my server went down for an hour when I clicked "Check users for spam" while there were over 100-200--500+ email addresses listed and not deleted yet. (There was a gap between my subscription renewal, so the spammy users built up for a while). The pattern seems to be having way more emails listed than you have concurrent SQL connections allowed, plus the forum already being busy.

Looking at the logs the server admin gave me, mysqld ran out of connections with massive numbers of "Waiting for table level lock" events (over 100 "Waiting for table level lock" piled up and clogged/deadlocked). For security reasons I can't publicly share the logs here, but a lot of deadlocked (waiting for table level lock) "INSERT INTO XXXXXXXbb_confirm" lines.

I could still ssh in so I found a workaround by "killall --user USERID" via ssh without needing to reboot the server (took a long time for me to figure this workaround out to kill all processes specific to the discussion forum, without needing to call my host sysadmin for emergency root access).

Then I clicked the "Check users for spam again" -- and right on cue (within 1-2 minutes) mysqld jammed completely, bringing down all mysql-dependant servers on my dedicated server.

Now I'll only check a few users at a time, but you might want to add some code to throttle the rate of queries to prevent unusual numbers of multithreaded "Waiting for table level lock" events.

If your forum has plenty of users, a lot of spammers, and you're a bit behind in executing a manual check (e.g. right after reactivating a subscription) -- I just wanted to let you know that Cleantalk phpbb plugin can cause 100% dedicated non-VPS Xeon server mysqld to go down for the count (without manual intervention) if you click "Check users for spam again" with over 100+ listed and keeping that textbox blank.

Temporary Workaround: Manually check only a few emails at a time -- and do it during off hours.

Permanent Workaround: Clealktalk phpbb plugin should code for gentler database access in a situation of being very far behind in spammer check in a "subscription-reactivation" manual-check situation.

[Alex]

Hello. Thank you. We will inspect this in a few days.

[Mark Rejhon]

Thanks.

I suspect it is probably hard to reproduce unless you let your subscription lapse sufficiently long enough on a very busy discussion forum.

[Artem Anoshin]

Thank you for your comment. This task will not be solved quickly, the approximate time is one month.

[Mark Rejhon]

Thanks.

I'm now resubscribed (for many years) and the backlog is gone, so it's not likely a sufficiently-risky backlog will happen to me again in the near future --

That being said.... it will likely impact other users who decide to re-subscribe after a long gap between subscriptions (the precise time you want to maintain a good impression).

BTW, normally the database queries self-throttle, but it felt like a possible race condition -- where 100+ simultaneous table locks were waiting for each other and created some kind of infinite loop or deadlock. (Keep in mind I had a full dedicated server, and also only 25% of RAM was used up). If this was a concurrent multithreaded forking of SQL queries that also had network calls (and thus slowed down query completion), maybe throttle it based on network calls (e.g. calls to cleantalk API), to prevent the table from being locked for longer than necessary while waiting for result from cleantalk servers. Just running off theories why I witnessed so many concurrent table locks waiting on each other. Hopefully these observation clues helps you find a quick fix, which will help other people in the future.

[SVFCode]

So, our tests show that the problem is local and related to database settings. No changes to the plugin are required.

[Mark Rejhon]

So, our tests show that the problem is local and related to database settings. No changes to the plugin are required.

I am unaware what my web hosting service did to my settings, but they claim it's configured in a way to retard DDoS/hacking attempts.

I haven't touched the settings of my server (I don't have root access even though it's a dedicated web server). I know how to navigate the command lines and configuration file, but this is "on the other side of the wall" of the Managed Dedicated Server (e.g. their responsibility)

Can you enlighten me on something I don't have control over?
I can advise them about it.

(In other words: HELP, I'm afraid to click the button again)

Thank you thank you 🙏🙏🙏