DNSError when providing nameservers to DNSResolver
maartendraijer opened this issue · comments
Hi!
Question. I'm trying to fetch the name servers of a specific domain name. This works when I don't provide nameservers to the DNSResolver
:
import asyncio
import aiodns
loop = asyncio.get_event_loop()
resolver = aiodns.DNSResolver(loop=loop)
f = resolver.query('test.de', 'NS')
result = loop.run_until_complete(f)
print(result)
This results in:
[ares_query_ns_result(host='dns3.iwelt-ag.de', ttl=None), ares_query_ns_result(host='dns2.iwelt-ag.net', ttl=None), ares_query_ns_result(host='dns.iwelt-ag.net', ttl=None)]
But, when I do this:
import asyncio
import aiodns
loop = asyncio.get_event_loop()
resolver = aiodns.DNSResolver(loop=loop, nameservers=['81.91.164.5'])
f = resolver.query('test.de', 'NS')
result = loop.run_until_complete(f)
print(result)
This is the result:
aiodns.error.DNSError: (1, 'DNS server returned answer with no data')
Though when I capture the packets I see that I'm getting the results back from the name server:
sudo tcpdump -i en0 -AAA -vvv host 81.91.164.5
tcpdump: listening on en0, link-type EN10MB (Ethernet), capture size 262144 bytes
21:40:30.518946 IP (tos 0x0, ttl 64, id 23865, offset 0, flags [none], proto UDP (17), length 53)
192.168.178.10.55500 > f.nic.de.domain: [udp sum ok] 58234+ NS? test.de. (25)
.Y3.;/..V?. ..E..5]9..@..k...
Q[.....5.!...z...........test.de.....
21:40:30.532423 IP (tos 0x0, ttl 57, id 27877, offset 0, flags [none], proto UDP (17), length 146)
f.nic.de.domain > 192.168.178.10.55500: [udp sum ok] 58234- q: NS? test.de. 0/3/1 ns: test.de. [1d] NS dns.iwelt-ag.net., test.de. [1d] NS dns2.iwelt-ag.net., test.de. [1d] NS dns3.iwelt-ag.de. ar: dns3.iwelt-ag.de. [1d] A 85.236.41.196 (118)
..V?. .Y3.;/..E...l...9..bQ[.....
.5...~...z...........test.de.............Q....dns.iwelt-ag.net.........Q....dns2.)........Q....dns3.iwelt-ag...V......Q...U.).
Am I missing something, or is this a bug somewhere in aiodns
or pycares
?
I added the following test in pycares: saghul/pycares@9689d86
It passes everywhere except on PyPy (for other reasons). So I guess it's the underlying c-ares library. Can you try to install or compile adig (included with c-ares, it's like dig but based on c-ares) and give it a try? Or try another nameserver just in case too?
Thanks for your reply @saghul.
Hmm, when I run this:
import asyncio
import aiodns
loop = asyncio.get_event_loop()
resolver = aiodns.DNSResolver(loop=loop, nameservers=['8.8.8.8'])
f = resolver.query('test.de', 'NS')
result = loop.run_until_complete(f)
print(result)
I get a result:
[ares_query_ns_result(host='dns3.iwelt-ag.de', ttl=None), ares_query_ns_result(host='dns2.iwelt-ag.net', ttl=None), ares_query_ns_result(host='dns.iwelt-ag.net', ttl=None)
These are the packets:
tcpdump: listening on en0, link-type EN10MB (Ethernet), capture size 262144 bytes
08:19:45.295169 IP (tos 0x0, ttl 64, id 43082, offset 0, flags [none], proto UDP (17), length 53)
192.168.50.68.60405 > google-public-dns-a.google.com.domain: [udp sum ok] 64917+ NS? test.de. (25)
...6....V?. ..E..5.J..@..q..2D.......5.!...............test.de.....
08:19:45.323672 IP (tos 0x0, ttl 48, id 41764, offset 0, flags [none], proto UDP (17), length 130)
google-public-dns-a.google.com.domain > 192.168.50.68.60405: [udp sum ok] 64917 q: NS? test.de. 3/0/0 test.de. [9m59s] NS dns3.iwelt-ag.de., test.de. [9m59s] NS dns2.iwelt-ag.net., test.de. [9m59s] NS dns.iwelt-ag.net. (102)
..V?. ...6....E....$..0..J......2D.5...n...............test.de..............W...dns3.iwelt-ag...........W...dns2.iwelt-ag.net..........W...dns.F
Though when I try any of these servers (from the ADDITIONAL SECTION
), it fails:
; <<>> DiG 9.8.3-P1 <<>> de @a.root-servers.net NS
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62080
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 6, ADDITIONAL: 10
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;de. IN NS
;; AUTHORITY SECTION:
de. 172800 IN NS a.nic.de.
de. 172800 IN NS f.nic.de.
de. 172800 IN NS l.de.net.
de. 172800 IN NS n.de.net.
de. 172800 IN NS s.de.net.
de. 172800 IN NS z.nic.de.
;; ADDITIONAL SECTION:
a.nic.de. 172800 IN A 194.0.0.53
f.nic.de. 172800 IN A 81.91.164.5
l.de.net. 172800 IN A 77.67.63.105
n.de.net. 172800 IN A 194.146.107.6
s.de.net. 172800 IN A 195.243.137.26
z.nic.de. 172800 IN A 194.246.96.1
a.nic.de. 172800 IN AAAA 2001:678:2::53
f.nic.de. 172800 IN AAAA 2a02:568:0:2::53
l.de.net. 172800 IN AAAA 2001:668:1f:11::105
n.de.net. 172800 IN AAAA 2001:67c:1011:1::53
;; Query time: 30 msec
;; SERVER: 198.41.0.4#53(198.41.0.4)
;; WHEN: Fri Jul 1 08:21:32 2016
;; MSG SIZE rcvd: 334
But what tcpdump
shows me is that results are coming back from any of them:
sudo tcpdump -i en0 -AAA -vvv host 194.246.96.1
tcpdump: listening on en0, link-type EN10MB (Ethernet), capture size 262144 bytes
08:27:39.786882 IP (tos 0x0, ttl 64, id 10852, offset 0, flags [none], proto UDP (17), length 53)
192.168.50.68.63445 > z.nic.de.domain: [udp sum ok] 55343+ NS? test.de. (25)
...6....V?. ..E..5*d..@.:p..2D..`....5.!.<./...........test.de.....
08:27:39.804282 IP (tos 0x0, ttl 55, id 32999, offset 0, flags [none], proto UDP (17), length 146)
z.nic.de.domain > 192.168.50.68.63445: [udp sum ok] 55343- q: NS? test.de. 0/3/1 ns: test.de. [1d] NS dns.iwelt-ag.net., test.de. [1d] NS dns2.iwelt-ag.net., test.de. [1d] NS dns3.iwelt-ag.de. ar: dns3.iwelt-ag.de. [1d] A 85.236.41.196 (118)
..V?. ...6....E.......7.....`...2D.5...~.../...........test.de.............Q....dns.iwelt-ag.net.........Q....dns2.)........Q....dns3.iwelt-ag...V......Q...U.).
And using dig
things are working too:
dig test.de NS @81.91.164.5
; <<>> DiG 9.8.3-P1 <<>> test.de NS @81.91.164.5
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64228
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 3, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;test.de. IN NS
;; AUTHORITY SECTION:
test.de. 86400 IN NS dns.iwelt-ag.net.
test.de. 86400 IN NS dns2.iwelt-ag.net.
test.de. 86400 IN NS dns3.iwelt-ag.de.
;; ADDITIONAL SECTION:
dns3.iwelt-ag.de. 86400 IN A 85.236.41.196
;; Query time: 38 msec
;; SERVER: 81.91.164.5#53(81.91.164.5)
;; WHEN: Fri Jul 1 08:31:14 2016
;; MSG SIZE rcvd: 118
Also I tried some other tlds (com & nl), and those are not working either. Nasty.
I can't seem to find anything about adig
. I'm on OS X by the way.
Hum, I really don't know what's going on :-( I can only speculate it's somewhere deep down c-ares, since using a different nameserver works...
@ioc32 hey mate, does any of this (the DNS traces) make any sense to you?
@saghul @maartendraijer I do not currently have a virtualenv where I can test this, but what you're experiencing may be related to the underlying subtleties in the DNS queries you are firing and how strict the library backend is when parsing them.
Let me elaborate:
DNS is an extremely lenient protocol and will allow for "some" answers being sent back as responses to queries that are not strictly correct.
In this case, you get the expected answers from aiodns/pycares when you pass a caching resolver to the DNSResolver class, but you do not when you pass an authoritative name-server (all name-servers in the additional section are of course authoritative).
I don't know how DNSResolver treats DNS responses with the RA (Recursion Available) bit set to zero.
Please have a look at the differences in the DNS flags and responses here [1], hopefully the examples there are more helpful than my explanation :)
[1]
iortiz@rigel:~
$ dig @81.91.164.5 +norec test.de ns
; <<>> DiG 9.8.3-P1 <<>> @81.91.164.5 +norec test.de ns
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19290
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 3, ADDITIONAL: 1
;; QUESTION SECTION:
;test.de. IN NS
;; AUTHORITY SECTION:
test.de. 86400 IN NS dns.iwelt-ag.net.
test.de. 86400 IN NS dns2.iwelt-ag.net.
test.de. 86400 IN NS dns3.iwelt-ag.de.
;; ADDITIONAL SECTION:
dns3.iwelt-ag.de. 86400 IN A 85.236.41.196
;; Query time: 29 msec
;; SERVER: 81.91.164.5#53(81.91.164.5)
;; WHEN: Fri Jul 1 12:10:32 2016
;; MSG SIZE rcvd: 118
iortiz@rigel:~
$ dig @81.91.164.5 test.de ns
; <<>> DiG 9.8.3-P1 <<>> @81.91.164.5 test.de ns
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32736
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 3, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;test.de. IN NS
;; AUTHORITY SECTION:
test.de. 86400 IN NS dns.iwelt-ag.net.
test.de. 86400 IN NS dns2.iwelt-ag.net.
test.de. 86400 IN NS dns3.iwelt-ag.de.
;; ADDITIONAL SECTION:
dns3.iwelt-ag.de. 86400 IN A 85.236.41.196
;; Query time: 30 msec
;; SERVER: 81.91.164.5#53(81.91.164.5)
;; WHEN: Fri Jul 1 12:10:36 2016
;; MSG SIZE rcvd: 118
iortiz@rigel:~
$ dig @8.8.8.8 test.de ns
; <<>> DiG 9.8.3-P1 <<>> @8.8.8.8 test.de ns
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14453
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;test.de. IN NS
;; ANSWER SECTION:
test.de. 599 IN NS dns2.iwelt-ag.net.
test.de. 599 IN NS dns.iwelt-ag.net.
test.de. 599 IN NS dns3.iwelt-ag.de.
;; Query time: 29 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Jul 1 12:10:49 2016
;; MSG SIZE rcvd: 102
btw we're hitting this too when used w/ aiohttp with requests to water.weather.gov:80
update: seems to be resolved
I added the following test in pycares: saghul/pycares@9689d86
It passes everywhere except on PyPy (for other reasons). So I guess it's the underlying c-ares library. Can you try to install or compile adig (included with c-ares, it's like dig but based on c-ares) and give it a try? Or try another nameserver just in case too?
could you try this on debian stretch with python3.6? I tried both the pycares wheel and installing from source.
After building c-ares from source@master, here are the results comparing dig with adig
bash-4.4# dig @81.91.164.5 test.de
; <<>> DiG 9.10.3-P4-Debian <<>> @81.91.164.5 test.de
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49319
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 3, ADDITIONAL: 2
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;test.de. IN A
;; AUTHORITY SECTION:
test.de. 86400 IN NS dns.iwelt-ag.net.
test.de. 86400 IN NS dns2.iwelt-ag.net.
test.de. 86400 IN NS dns3.iwelt-ag.de.
;; ADDITIONAL SECTION:
dns3.iwelt-ag.de. 86400 IN A 85.236.41.196
;; Query time: 192 msec
;; SERVER: 81.91.164.5#53(81.91.164.5)
;; WHEN: Wed Nov 14 23:13:23 UTC 2018
;; MSG SIZE rcvd: 129
bash-4.4# bin/adig -s 81.91.164.5 -t NS test.de
DNS server returned answer with no data
id: 10118
flags: qr rd
opcode: QUERY
rcode: NOERROR
Questions:
test.de . NS
Answers:
NS records:
test.de . 86400 NS dns.iwelt-ag.net.
test.de . 86400 NS dns2.iwelt-ag.net.
test.de . 86400 NS dns3.iwelt-ag.de.
Additional records:
dns3.iwelt-ag.de. 86400 A 85.236.41.196
Closing since there is not much we can do on our side. c-ares is doing the heavy lifting. That said, pycates 3.0.0 (currently in beta) bumped c-ares to version 1.15.0, maybe some bug related to this got fixed.