Accessing the site doesn't work 503 Service Unavailable error
GoWithTheFlow12 opened this issue · comments
Problem
Evidence / Screenshot
Relevant URL(s)
When I access the website it loads very slow, and when it loads I have this error:
503 Service Unavailable
No server is available to handle this request.
Reproducing the bug
- Go to ...https://openlibrary.org
- Do ...just wait for the page to load, it does not load
- Expected behavior:
- Actual behavior:
Context
- Browser (Chrome, Safari, Firefox, etc): tested on chrome and mozzila
- OS (Windows, Mac, etc): Windows
- Logged in (Y/N): I was logged in when the error occured, then I tested in other browser where I was not logged in and it didn't worked as well
- Environment (prod, dev, local): prod
Notes from this Issue's Lead
Proposal & constraints
Related files
Stakeholders
Note: Before making a new branch or updating an existing one, please ensure your branch is up to date.
Unfortunately archive.org is experiencing a DDOS attack; you can monitor the state here: https://x.com/internetarchive/status/1795117949499445554?t=zqku2Aj-neWfjdCruTvCgw
Hi people,
This is CC attack. Could I help with this?
Please provide me contact person - so I can share ways to stop this kind of attack
I did have the same attack, then I investigated how to stop it - I also tested against the tools they are using.
Most of IP are proxy servers and you have full log of this:
"GET /?search=AiMEppijD"
So when you get this msg - contact me I am willing to help ASAP providing a script to "patch" against this attack.
The problem has been resolved! Thanks for the offer @vladimircicovic ! We have staff who help resolve issues like these, but I'll forward your message along 👍
Again you have DDOS but this time it looks like a UDP flood or ICMP flood (I am guessing this, based on the ping missing packet)
Also, I could help with this new attack. For the record your https://archive.org/, - if not anycast (same IP around the globe) and it is one IP so they could flood with easily. The strategy for this would be to use Point of Presence around the globe.
Limiting ICMP/UDP packets per second with your firewall could also help in this case.
EDIT: I checked it, and there is no anycast, it is only one IP at gw internet-archive.e0-29.core2.sfo1.he.net
So when they attack - failure will go on the part of the PoP IP and not all PoPs you have.
They change tactics as you make updates.
ping archive.org
PING archive.org (207.241.224.2) 56(84) bytes of data.
64 bytes from www.archive.org (207.241.224.2): icmp_seq=2 ttl=51 time=175 ms
^C
--- archive.org ping statistics ---
8 packets transmitted, 1 received, 87.5% packet loss, time 7101ms
rtt min/avg/max/mdev = 175.214/175.214/175.214/0.000 ms
ping -c 5 archive.org
PING archive.org (207.241.224.2) 56(84) bytes of data.
--- archive.org ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4045ms
Hi @vladimircicovic thanks for the help! Yeah there's another event occurring now: Updates here https://x.com/internetarchive/status/1795451463465845141?s=46&t=zqku2Aj-neWfjdCruTvCgw
Hi @vladimircicovic thanks for the help! Yeah there's another event occurring now: Updates here https://x.com/internetarchive/status/1795451463465845141?s=46&t=zqku2Aj-neWfjdCruTvCgw
My last comment was for that attack. Before was an HTTP layer 7 attack, now is (was) probably an ICMP or UDP flood attack.
@vladimircicovic thank you very much for sharing your insights and helping us. It's greatly appreciated. Would you be open to an invite to our internetarchive / openlibrary slack in case we have followup questions? Thank you again
cc: @bfalling, closing this issue for now
@vladimircicovic: One option our security team is considering would be a synproxy recipe for linux ipvs in Direct Routing mode, if we can find one. Is this something you would know about?
@vladimircicovic: One option our security team is considering would be a synproxy recipe for linux ipvs in Direct Routing mode, if we can find one. Is this something you would know about?
Let us connect on Slack and I will share information with the team
@vladimircicovic thank you very much for sharing your insights and helping us. It's greatly appreciated. Would you be open to an invite to our internetarchive / openlibrary slack in case we have followup questions? Thank you again
cc: @bfalling, closing this issue for now
Yes, invite me