Only (maximum) last 100 entries get into .xml
stienheremans opened this issue · comments
Dear,
I have tried to use the script for downloading all sen2 images over Belgium.
However, since the .xml file is cut after 100 entries (0 to 99), it never searches any further back into time.
The parser mentions the '-r' option as number of records to download, but actually it refers to the number of records to put into the xml, causing far less records to actually download.
Is it possible to remove the constraint on the number of records in the xml?
Kind regards,
Stien
Dear Stien,
I'll try to find some time to look at it. I have heard that ESA has changed things regarding the number of records we can get at a time to avois the overload of its servers it is experiencing these days. Anyway, I guess you still might use the selection by date to get the older results.
Best regards,
Olivier
You are right Stien, We are not able anymore to use -r xxx with xxx>100 , while -r 99 works. -r 101 throws an Error 500, Internal Server Error. This is a new limitation added by ESA.
And as the selection by date is done a posteriori on the product references, it does not help. I once had an a priori selection by date, but it only corresponded to the production date and not to the acqusition date. Would it still be useful for you ? I can put it back.
Otherwise, I guess I'll have to find a workaround.
Olivier
You may now use queries with constraints on ingestion date
python Sentinel_download.py --lat 43.6 --lon 1.44 -a apihub.txt -o 51 --id 2016-11-01 --if 2016-12-01
I created a fix (see pull request), you can now set -r to values larger than 100. It will now:
- look for every download available download in steps of 100
- download the necessary files in these 100 downloads
- repeat above two steps until downloads == the value set at -r (default 100)
Looks great, thanks a lot Gert !
I will merge both versions, but I am not sure I will test it fully right on (need to download > 100 products...)