This downloader package is part of the EU Interreg Project: Alpine Drought Observation
Package documentation is available at readthedocs.
ado-downloader can be installed using the following command:
python setup.py install
For detailed installation instructions, especially how to install dependencies, please refer to the INSTALL section of the documentation.
This submodule automates big data downloads from the Copernicus Climate Data Store. Based on the cdsapi package, it enables us to automatically chunk big download requests into parallel processed sub requests.
The downloader is based on the cdsapi and therefore also uses its authentication method. In order to use the downloader, one has to create a file with user credentials api-how-to in the home directory. Alternatively, define two environment variables 'CDSAPI_URL' and 'CDSAPI_KEY' with the same user credentials. Either way, one needs a cds.climate.copernicus.eu user account in order to use the downloader.
One way to use the cds_downloader is by copying the API request from the cds product page and create the Downloader object with the classmethod from_cds.
To create a Downloader object, e.g. from the example image above, one can easily copy the two arguments marked in black.
from cds_downloader import Downloader
test_downloader = Downloader.from_cds(
'reanalysis-era5-single-levels',
{
'product_type': 'reanalysis',
'format': 'grib',
'variable': '2m_temperature',
'year': [
'1979', '1980',
],
'month': [
'01', '02',
],
'day': [
'01', '02',
],
'time': [
'00:00', '01:00',
],
}
)
test_downloader.get_data("storage_path", ["year", "month"])To retrieve the data with cdsapi one has to call the method get_data.
The maximum single data request size depends on the copernicus climate data store and is automatically extracted from their metadata webapi.
If no split_keys are specified, the method automatically chunks the cds request into multiple smaller requests and spawns a single process for each of them. Therefore, it extracts all list-like objects from the cds_filter (e.g. "year", "month", ...) and splits the data into single requests/files.