Not all domains expiry date is exported, but whois does show expiry info
maintain3r opened this issue · comments
Hello,
I want to use your project to monitor a list of corporate domains.
I put the exporter in place and made a grafana dashboard, everything looks good except for some domains.
There's no expiry data on some of the domains, but interestingly, linux whois retrns expiry data.
I assume this project also uses whois tool to extract dns domain data and I was wondering if you can fix it.
p.s. I understand that there're some domains that hide their data on purpose, but why would linux whois work then.
Im excited to see this tool in prod.
Thanks for looking into this.
Which domains are not working?
Where are they registered?
Here's an example
$ whois 227359.biz | grep -i expi
Registry Expiry Date: 2024-05-05T23:59:59Z
I use docker img from docker hub, should I recompile the img to have the latest version of whois pkg?
It could be nice if there's a way to set a delay between whois calls as I get throttled by whois providers.
Something like
domains:
- google.com
- name: reddit.com
host: whois.godaddy.com
options:
rtimeout: 3s
rdelay: 10ms
This way I can control the throttling behaviour.
The rtimeout option would help in cases where whois cmd hangs (yes I see it's happening)
The rdeay option is even more important than rtimeout as it should give me a better control over throttling
Thanks again!
domain_exporter does not use the whois binary from the system, it uses a library to make the whois requests.
Here's an example
$ whois 227359.biz | grep -i expi
Registry Expiry Date: 2024-05-05T23:59:59Z
I don't get this result here:
❯ whois 227359.biz | grep -i expi
NOTICE: The expiration date displayed in this record is the date the
currently set to expire. This date does not necessarily reflect the expiration
view the registrar's reported date of expiration for this registration.
Ok, is the expiry data showed by domain_exporter accurate? And is there a way to get that domain info using exporter?
Thanks
the data comes from whois, just not the whois binary. It is accurate. If your registrar does not export the expiry date, not much we can do, if it does, it should work.
How about throttling?
When I put the entire list of my domains I get the following error for some domains
error="failed to fetch whois request: dial tcp: lookup whois.enom.com: i/o timeout"
If I put just a few domains in the config, everything works fine.
Looks like exporter does not wait until all data is returned back and closes the session.
Could you please add a delay parameter so that we can be less aggressive when we poke whois providers?
a delay would not work I think, will probably need to do some sort of backoff in the clients...
Here's a piece of log that I see in my docker container, I have a few dozens of domains in my config and it looks like exporter does not wait until a response is received.
error="failed to fetch whois request: dial tcp: lookup whois.godaddy.com: i/o timeout"
error="failed to fetch whois request: dial tcp: lookup whois.godaddy.com: i/o timeout"
error="failed to fetch whois request: dial tcp: lookup whois.godaddy.com: i/o timeout"
error="failed to fetch whois request: dial tcp: lookup whois.godaddy.com: i/o timeout"
error="failed to fetch whois request: dial tcp: lookup whois.godaddy.com: i/o timeout"
Any plans to fix this? I would be very unfortunate if not cus the idea of this exporter is quite useful.
Thanks!
added a --timeout flag and also increased the default timeout from 10s to all domains to 10s * number of domains.
Hi Carlos,
Thanks for looking into this problem.
It looks like the timeout issues comes from the fact that some requests are getting throttled.
Please take a look
error="could not parse whois response: "Rate limit exceeded. Try again after: 0s\r\n
error="could not parse whois response: "Rate limit exceeded. Try again after: 0s\r\n
error="could not parse whois response: "Rate limit exceeded. Try again after: 0s\r\n\
And since the whois provider rate limits the client, the client waits up until 30s and then times the connection out. So basically the connection timed out issue is just a consequence that comes from the throttling.
Could you please take a look, maybe there's a way to impose a sleep somewhere between each call to whois provider?
Thanks!
a sleep wont matter, needs to be a exponential backoff. I'm swamped now, but feel free to pr it.
We just started to have the similar problem. All .com domains we are monitoring show error, all other TLDs are okay. We thought our IP was banned for some reason but the rate limit makes sense. It started happening on August 1st.
just merged the timeout settings, can you try?
I have patched our domain exporter and set the timeout to 10 minutes, we have a prometheus set to try it every few hours so we'll see if it will pass through the next time it will run.
thanks!
Increasing the timeout is not a solution, there should be a retry implemented, so that if you get an error from the whois provider you retry in the delay specified in the error:
error="could not parse whois response: "Rate limit exceeded. Try again after: 58s\r\n\
or retry in a specific order like 10 times in total each time increasing the delay between attempts
10s
20s
30s
40s
...
Ok, the timeout has helped, but as @maintain3r says, retrying with increasing delay would be a more correct solution.