[datasets] Allow parallel processing of downloads
frgfm opened this issue Β· comments
π Feature
Use parallel processing for dataset downloads when a list of URLs is involved.
Motivation
pyronear.datasets.OpenFire
downloading process is taking several minutes while allowing multi-processing would shorten this by a lot.
Pitch
Add a multi-processing feature for URL downloads and let the users pick the number of workers. Potential libraries to check: Ray, python multiprocessing
A good implementation for URL downloading is already present in fastai
(but it is a large dependency).