pyronear / pyro-vision

Computer vision library for wildfire detection 🌲 Deep learning models in PyTorch & ONNX for inference on edge devices (e.g. Raspberry Pi)

Home Page:https://pyronear.org/pyro-vision/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[datasets] Allow parallel processing of downloads

frgfm opened this issue Β· comments

πŸš€ Feature

Use parallel processing for dataset downloads when a list of URLs is involved.

Motivation

pyronear.datasets.OpenFire downloading process is taking several minutes while allowing multi-processing would shorten this by a lot.

Pitch

Add a multi-processing feature for URL downloads and let the users pick the number of workers. Potential libraries to check: Ray, python multiprocessing
A good implementation for URL downloading is already present in fastai (but it is a large dependency).