ipv4.icanhazip.com timing out a lot
avivek opened this issue · comments
For the last few hours pings to ipv4.icanhazip.com has started timing out.
@avivek My monitoring has been going off as well. I'm working to figure out the cause.
@avivek Looks like an IPv4-based DDoS attack. These do happen from time to time and I'm trying to mitigate this one if I can.
@avivek Some adjustments made -- looking better now? My monitoring is green again.
HTTPS is still timing out for me: https://ipv4.icanhazip.com
HTTP does work though: http://ipv4.icanhazip.com
Ugh, the DDoS sources are changing quite frequently. Still working through some changes.
HI @major
Thanks a lot for being responsive about this. Really appreciate that.
Though the time out frequency has reduced quite a bit now, it does happen rarely.
@avivek i'm in the process of moving back to CloudFlare workers for now. This will mess up the ipv4/ipv6 DNS records, but I can't do much about this DDoS at the current size.
@major, just to confirm, the expected messing-up of the DNS records has us resolving via AAAA and connecting via IPv6 to ipv4.icanhazip.com in some cases, which unexpectedly returns the client IPv6 address. Is there a timeline for the AAAA record being removed from the ipv4 domain (the HTTP library we're using doesn't support controlling IPv4 vs IPv6 routing without doing some nasty patching)?
@haikuginger Yes, my hands are tied on this one, unfortunately. When you proxy through Cloudflare, every DNS record gets a v4/v6 address assigned no matter what. I've raised this to CloudFlare a few times in the part year and they are looking at some potential solutions. At the moment, the only workaround I know of is to explicitly specify the network stack, like curl -4 icanhazip.com, when you make the request. I'm trying to see if I can get creative to work around this, but I haven't been able to figure something out yet.
If anyone has ideas, I am all ears!
So when I check DNS records:
➜ host ipv4.icanhazip.com
ipv4.icanhazip.com has address 104.22.19.188
ipv4.icanhazip.com has address 172.67.9.138
ipv4.icanhazip.com has address 104.22.18.188
ipv4.icanhazip.com has IPv6 address 2606:4700:10::6816:13bc
ipv4.icanhazip.com has IPv6 address 2606:4700:10::ac43:98a
ipv4.icanhazip.com has IPv6 address 2606:4700:10::6816:12bc
There could be a way to flip ipv4.icanhazip.com to DNS Only in Cloudflare's portal and point it to those v4 addresses manually. I don't know if this actually works nor do I know how often Cloudflare changes these addresses. 🤔
The manual setting causes breakage because the static files (like robots.txt are no longer available). 😞
Also, the IPv4/IPv6 addresses change relatively frequently.
@major I wonder if updating icanhaz-app.conf to conditionally have <VirtualHost 0.0.0.0:80> instead of *:80 would restrict httpd to listen only on IPv4? That way the DNS might still have an issue, but well-behaved clients should fall back to IPv4 when they can't establish a connection.
If that doesn't work, the icanhaz-config.stub might need to be changed instead to specifically have its Listen directives modified similarly (although the default to Listen on all 80 might need to be overwritten somehow...not sure.)
@haikuginger @avivek Good news! The fine people at Cloudflare sent over a bucketload of instructions so I can get the DNS records fixed properly. It takes some interesting workarounds, but it should be something that should work well. I'll work through those today.
@haikuginger @avivek Could y'all please test:
It looks like there's still an issue on the ipv6 side. I have some questions that I sent over to the Cloudflare folks about how to fix it.
➜ host ipv6.icanhazip.com rustam.ns.cloudflare.com
Using domain server:
Name: rustam.ns.cloudflare.com
Address: 2606:4700:58::a29f:2c94#53
Aliases:
ipv6.icanhazip.com has address 104.21.34.198
ipv6.icanhazip.com has address 172.67.164.120
ipv6.icanhazip.com has IPv6 address 2606:4700:3030::6815:22c6
ipv6.icanhazip.com has IPv6 address 2606:4700:3034::ac43:a478
Thanks to everyone for being patient. 🤗
@major I'm seeing only A records when I do the DNS lookup, so I think we should be good to go. I don't myself have full IPv6 connectivity, so I'm asking my users who do to recheck that our use case is working again, now that the AAAA records have been removed.
@major after checking with my colleagues I think I can say that our use case is solved for. Thanks for the rapid turnaround on this, and the service in general!
@haikuginger Happy to help!