Cadair / parfive

An asyncio based parallel file downloader for Python 3.8+

Home Page:https://parfive.readthedocs.io/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Ipython terminal breaks

nabobalis opened this issue · comments

from sunpy.net import Fido, attrs as a
import astropy.units as u
from os import mkdir, path

results = Fido.search(a.jsoc.Time('2019/01/01', '2019/02/01'), a.jsoc.Notify('nabil.freij@gmail.com'), a.jsoc.Series('aia.lev1_euv_12s'), a.jsoc.Wavelength(304 * u.angstrom),a.Sample(2 * u.hour))

downloaded_file = Fido.fetch(results, path= path.abspath('AIA304')+'/{file}')

The downloads start, the progress bar shows up and then it kind of dies.

Untitled

Adding this is a workaround

num_missing = 100
while num_missing > 0:
    downloaded_file = Fido.fetch(results, path= path.abspath('AIA304')+'/{file}')
    num_missing = len(downloaded_file.errors)

Untitled_2

I wonder if this is purely parfive or more fido and parfive.

Turns out it happens in a normal python terminal too: See sunpy/sunpy#4555

Since this seems to be a "number of connections" issue, I think this can be closed in favour of sunpy/sunpy#4555.

This shouldn't be happening though even if the downloads fail.

Or maybe it should be.

What shouldn't be happening? I thought parfive is silent on failed downloads by design?

oh yeah maybe this is actually expected, but I was looking at the wrong screenshot.

Perhaps it should print something at the end saying "not all downloads completed" or similar

cough #43 cough

yep that one 😀