HTTP request sent, awaiting response... 502 Bad Gateway

[Ciaran]

Hi there,

Great script:

My cmd:

ubuntu@ubuntu:~$ Sentinel_download.py --latmin 0.904282606986 --latmax -0.90428260986 --lonmin 34.797275229 --lonmax 36.592770943 -a '/home/ubuntu/anaconda3/bin/apihub.txt' -m 20 --start_date 20151010 --end_date 20161230 --tile 36MZE -w /media/ubuntu/HD-PZFU3/Kenya/S2_training/L1C_tiles_mau/ --dhus -r 200

It seems to run fine but exits eventually with this error (although it has successfully downloaded plenty granules).

HTTP request sent, awaiting response... 502 Bad Gateway 2016-08-16 16:00:09 ERROR 502: Bad Gateway

Traceback (most recent call last): File "/home/ubuntu/anaconda3/envs/python2/bin/Sentinel_download.py", line 361, in <module> get_dir('DATASTRIP',url_datastrip_dir,product_dir_name,wg,auth,wg_opt,value) File "/home/ubuntu/anaconda3/envs/python2/bin/Sentinel_download.py", line 67, in get_dir download_tree(product_dir_name+'/'+dir_name,"temp.xml",wg,auth,wg_opt,value) File "/home/ubuntu/anaconda3/envs/python2/bin/Sentinel_download.py", line 50, in download_tree download_tree(nom_rep,'files.xml',wg,auth,wg_opt,value) File "/home/ubuntu/anaconda3/envs/python2/bin/Sentinel_download.py", line 40, in download_tree urls,types,names=get_elements(xml_file) File "/home/ubuntu/anaconda3/envs/python2/bin/Sentinel_download.py", line 29, in get_elements line=fic.readlines()[0].split('<entry>') IndexError: list index out of range
`
This happens every time it runs. Do you know why this may be? I boosted the limit on number of images to download just in case although I admit this is not likely related to the error.

Thanks

[Olivier Hagolle]

Hi,
yes, it happens from time to time (not only to you ;) ), and seems to be caused by the ESA server.
I have just tried to trap the error (because at the beginning, the file was ignored, but was missing when one tried to use the tile).

Another solution could be to try again the file download. But how many times...
Olivier

[Gert Sterenborg]

After downloading 64 tiles, I did not encounter a problem when using:

while os.path.getsize( [path to downloaded file] )==0 :
    os.system(commande_wget)

I just added this after every os.system(commande_wget). Every time I have had 502 Bad Gateway the first retry succeeded. It would probably a good idea to use something like this:

max_tries = 3

for i in range(max_tries):
    os.system(commande_wget)
    if os.path.getsize( [path to downloaded file] ) > 0:
        # download succeeded
        break

This way I can be more confident about the script will succeed.

[Olivier Hagolle]

Good idea, i have generalized adding

while os.path.getsize( [path to downloaded file] )==0 :
    os.system(commande_wget)