Cadair / parfive

An asyncio based parallel file downloader for Python 3.8+

Home Page:https://parfive.readthedocs.io/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Downloads fail when behind proxy

wtbarnes opened this issue · comments

Any downloads behind a proxy hang indefinitely.

I realize this is difficult to test so I'm happy to be the guinea pig for any proposed fixes! Of course, all proxies may be slightly different as well...

MWE

>>> from parfive import Downloader
>>> dl = Downloader()
>>> dl.enqueue_file("http://data.sunpy.org/sample-data/predicted-sunspot-radio-flux.txt", path="./")
>>> files = dl.download()  # <-- this hangs forever

Also true when downloading sample data in sunpy or doing any query with sunpy.net.Fido.

It appears I should've just looked at the aiohttp docs first. As noted here, it looks like you can just pass a url to the session with proxy='my-proxy-url'. Presumably this would happen here?

The main issue of course is how best to propagate this configuration option...

Ok this is solved by just passing proxy='my-proxy-url' to enqueue_file so not actually an issue. However, this is likely a problem downstream in sunpy as I'm not sure we allow kwargs to be passed to enqueue_file.

I think we should support proxy env vars, and maybe also tie it into #32 and generally allow passing session vars to the constructor of Downloader().

@Cadair should I start working on this?

Now that #32 has been merged, can this be closed?