Segmentation fault upon application termination when querying lots and lots of DNS servers using separate DNSResolver instances

[Arvid Fahlström Myrman]

I'm not sure if the problem is in aiodns, pycares or even Python itself, but to start off this seems like a good place to put this, especially if it turns out that the issue is just me doing something very very wrong.

I'm trying to look up all IP addresses belonging to a certain hostname. Due to load balancing techniques different DNS servers may report different IP addresses, which means that I first need to find a list of public DNS servers, and then query each one with the hostname. From what I can tell, DNSResolver (or rather pycares) doesn't get very happy when you try to change the nameservers from different simultaneously running coroutines (not very surprising really), so separate DNSResolver instances are needed.

Here's an example of what I'm trying to do, using aiodns in conjunction with aiohttp to fetch the DNS server list (note that it may take several seconds to fetch the DNS list):

import asyncio
import json

import aiodns
import aiohttp

json_url = "http://public-dns.tk/nameservers.json"
address = "google.com"

@asyncio.coroutine
def main():
    try:
        print("Fetching DNS server list...")
        response = yield from asyncio.wait_for(
            aiohttp.request('GET', json_url), 30)
    except asyncio.TimeoutError:
        print("Error: Couldn't fetch DNS server list")
        return

    body = yield from response.read_and_close()
    dns_data = json.loads(body.decode('utf-8'))
    dns_servers = [entry['ip'] for entry in dns_data
                   if entry['state'] == 'valid'
                      and len(entry['ip'].split('.')) == 4]

    ips = set()
    i = 0
    @asyncio.coroutine
    def do_lookup(ip):
        nonlocal i
        try:
            resolver = aiodns.DNSResolver(nameservers=[ip],
                                          timeout=3, tries=2)
            ips.update((yield from resolver.query(address, 'A')))
        except Exception as e:
            print("Warning: Couldn't connect to DNS server {}"
                    .format(ip))
        i += 1
        print("Queried DNS server {}/{}".format(i, len(dns_servers)))

    print("Resolving IP addresses...")
    yield from asyncio.wait([do_lookup(server)
                             for server in dns_servers])
    print("Got IP addresses:", ips)


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

When running the above code I get the following output:

Fetching DNS server list...
Resolving IP addresses...
Warning: Couldn't connect to DNS server 85.185.171.82
Queried DNS server 1/3130
[...]
Queried DNS server 2205/3130
Queried DNS server 2206/3130
Queried DNS server 2207/3130
Queried DNS server 2208/3130
Queried DNS server 2209/3130
[...]
Warning: Couldn't connect to DNS server 207.172.11.73
Queried DNS server 3129/3130
Warning: Couldn't connect to DNS server 83.238.39.254
Queried DNS server 3130/3130
Got IP addresses: {'212.39.82.177', '74.125.239.99', [...], '173.194.124.39', '93.191.15.103'}
Segmentation fault

Backtrace:

Program received signal SIGSEGV, Segmentation fault.
visit_decref (op=0x7ffff017c370, data=0x0) at Modules/gcmodule.c:373
373     Modules/gcmodule.c: No such file or directory.
(gdb) bt
#0  visit_decref (op=0x7ffff017c370, data=0x0) at Modules/gcmodule.c:373
#1  0x00007ffff79d279b in list_traverse (o=0x7ffff2b29e48, visit=0x7ffff7a95ee0 <visit_decref>, arg=0x0)
    at Objects/listobject.c:2217
#2  0x00007ffff7a951bf in subtract_refs (containers=<optimized out>) at Modules/gcmodule.c:398
#3  collect (generation=generation@entry=2, n_collected=n_collected@entry=0x0, 
    n_uncollectable=n_uncollectable@entry=0x0, nofail=nofail@entry=1) at Modules/gcmodule.c:969
#4  0x00007ffff7a96301 in _PyGC_CollectNoFail () at Modules/gcmodule.c:1638
#5  0x00007ffff7a705f8 in PyImport_Cleanup () at Python/import.c:483
#6  0x00007ffff7a7cd16 in Py_Finalize () at Python/pythonrun.c:616
#7  0x00007ffff7a9408f in Py_Main (argc=-136464988, argv=0x0) at Modules/main.c:771
#8  0x0000000000400af6 in main ()

I'm using aiodns 0.3.0 with Python 3.4.1 running on 64-bit (Arch) Linux.

[Arvid Fahlström Myrman]

I should add that if I just use the first ~10 or so DNS servers, I generally don't see a segfault. As I increase the number of servers that I query simultaneously the probability of seeing a segfault increases.

[Saúl Ibarra Corretgé]

Thanks for the report! I see you are using the nameservers option on pycares.Channel. There was a refcounting bug there (saghul/pycares@b74412f) which is not yet on any released version. Can you please test latest pycares master and see if it doesn't crash? If so I'll make a pycares bugfix release.

[Arvid Fahlström Myrman]

Right, upgrading to master is probably the first thing I should've tried, sorry. Upgrading does seem to fix the issue from what I can tell.

[Saúl Ibarra Corretgé]

No worries! Thanks for confirming, I'll make a new pycares release soon then.

[Saúl Ibarra Corretgé]

Hey @BeholdMyGlory I just released pycares 0.6.2, which includes that fix. Thanks for the report!

[Arvid Fahlström Myrman]

Awesome, thanks for the release. Everything seems to be working as it should.