query_results.xml not complete
Opadera opened this issue · comments
Hello,
i receive the following error when running
python Sentinel_download.py --lat 43.6 --lon 1.44 -a apihub.txt -o 51 -s S2 --downloader aria2
aria2c --check-certificate=false --http-user="guest" --http-passwd="guest" --continue -o query_results.xml "https://scihub.copernicus.eu/apihub/search?q=footprint:"Intersects(43.600000,1.440000)" filename:S2*R051*&rows=100"
07/25 08:29:12 [ERROR] CUID#7 - Download aborted. URI=https://scihub.copernicus.eu/apihub/search?q=footprint:Intersects(43.600000,1.440000)%20filename:S2*R051*&rows=100
Exception: [AbstractCommand.cc:350] errorCode=22 URI=https://scihub.copernicus.eu/apihub/search?q=footprint:Intersects(43.600000,1.440000)%20filename:S2*R051*&rows=100
-> [HttpSkipResponseCommand.cc:239] errorCode=22 The response status is not successful. status=500
07/25 08:29:12 [NOTICE] Download GID#fec8660e908f0856 not complete: D:/ohagolle/query_results.xml
Download Results:
gid |stat|avg speed |path/URI
======+====+===========+=======================================================
fec866|ERR | 0B/s|D:/ohagolle/query_results.xml
Status Legend:
(ERR):error occurred.
aria2 will resume download if the transfer is restarted.
If there are any errors, then see the log file. See '-l' option in help/man page for details.
Traceback (most recent call last):
File "Sentinel_download.py", line 230, in <module>
xml=minidom.parse("query_results.xml")
File "C:\hagelprogramme\Anaconda\lib\xml\dom\minidom.py", line 1918, in parse
return expatbuilder.parse(file)
File "C:\hagelprogramme\Anaconda\lib\xml\dom\expatbuilder.py", line 922, in parse
fp = open(file, 'rb')
IOError: [Errno 2] No such file or directory: 'query_results.xml'
any ideas hwo to solve this?
Win10
I have exactly the same problem. The query_results.xml does not get generated and if i create it manually it deletes it and doesn't replace it.
`07/30 19:15:48 [ERROR] CUID#7 - Download aborted. URI=https://scihub.copernicus.eu/apihub/search?q=footprint:Intersects(-22.200000,-68.500000)%20filename:S1*&rows=100
Exception: [AbstractCommand.cc:350] errorCode=22 URI=https://scihub.copernicus.eu/apihub/search?q=footprint:Intersects(-22.200000,-68.500000)%20filename:S1*&rows=100
-> [HttpSkipResponseCommand.cc:239] errorCode=22 The response status is not successful. status=500
07/30 19:15:48 [NOTICE] Download GID#e646280060448dd4 not complete: G:/query_results.xml
Download Results:
gid |stat|avg speed |path/URI
======+====+===========+=======================================================
e64628|ERR | 0B/s|G:/query_results.xml
Status Legend:
(ERR):error occurred.
aria2 will resume download if the transfer is restarted.
If there are any errors, then see the log file. See '-l' option in help/man page for details.
Traceback (most recent call last):
File "Sentinel_download.py", line 230, in
xml=minidom.parse("query_results.xml")
File "C:\Python27\lib\xml\dom\minidom.py", line 1918, in parse
return expatbuilder.parse(file)
File "C:\Python27\lib\xml\dom\expatbuilder.py", line 922, in parse
fp = open(file, 'rb')
IOError: [Errno 2] No such file or directory: 'query_results.xml'`
What i have discovered is that the URLs in the command window doen have the "" around the Intersects.
This is what is printed, and if you go to this link you get a Error 500
https://scihub.copernicus.eu/apihub/search?q=footprint:Intersects(-22.200000,-68.500000)%20filename:S1*&rows=100
What it should be is this
My programming skills are rubbish and i cant work out how to change this in the Sentinel_download.py to test if this is the error or not.
Dear Unnic and Burl,
I do not use windows and I am not able to maintain this version. The windows version was contributed by a colleague. I am looking for someone who could maintain it. Unnic maybe ?
Depending on the system (linux, windows, apple), and the downloader (wget, aria2), you need to "protect" some of the characters. For instance, replace " by " or even "
This happens between lines 194 and 204.
If someone can try and tell me, it would be useful.
Olivier
Ok this seems to work for me. As mentioned previously, i am not a programming guy. I cant even program my microwave! However, this is what i changed at line 194
if geom=='point':
if sys.platform.startswith('linux') or sys.platform.startswith('darwin'):
query_geom='footprint:"Intersects(%f,%f)"'%(options.lat,options.lon)
else :
query_geom='footprint:"Intersects(%f,%f)"'%(options.lat,options.lon)
elif geom=='rectangle':
if sys.platform.startswith('linux') or sys.platform.startswith('darwin'):
query_geom='footprint:"Intersects(POLYGON(({lonmin} {latmin}, {lonmax} {latmin}, {lonmax} {latmax}, {lonmin} {latmax},{lonmin} {latmin})))"'.format(latmin=options.latmin,latmax=options.latmax,lonmin=options.lonmin,lonmax=options.lonmax)
else :
query_geom='footprint:"Intersects(POLYGON(({lonmin} {latmin}, {lonmax} {latmin}, {lonmax} {latmax}, {lonmin} {latmax},{lonmin} {latmin})))"'.format(latmin=options.latmin,latmax=options.latmax,lonmin=options.lonmin,lonmax=options.lonmax)
Basically to get the " before and after the Intersects phrase, i added \" at both the start and end.
This successfully generates the query_results.xml file and starts to process the file download. However, there is another error that is occurring which shows a doubling up of the file path.
07/31 18:05:57 [ERROR] CUID#7 - Download aborted. URI=https://scihub.copernicus.eu/apihub/odata/v1/Products('c67759c0-2db7-4d75-bc99-39e258bfbe30')/Nodes('S2A_OPER_PRD_MSIL1C_PDMC_20160703T230548_R096_V20160703T144853_20160703T144853.SAFE')/Nodes('rep_info')/Nodes('S2_User_Product_Level-1C_Metadata.xsd')/$value
Exception: [AbstractCommand.cc:402] errorCode=18 URI=https://scihub.copernicus.eu/apihub/odata/v1/Products('c67759c0-2db7-4d75-bc99-39e258bfbe30')/Nodes('S2A_OPER_PRD_MSIL1C_PDMC_20160703T230548_R096_V20160703T144853_20160703T144853.SAFE')/Nodes('rep_info')/Nodes('S2_User_Product_Level-1C_Metadata.xsd')/$value
-> [RequestGroup.cc:748] errorCode=18 Download aborted.
-> [util.cc:1584] errNum=22 errorCode=18 Failed to make the directory G://G:/test/S2A_OPER_PRD_MSIL1C_PDMC_20160703T230548_R096_V20160703T144853_20160703T144853.SAFE/rep_info, cause: Invalid argument
07/31 18:05:57 [NOTICE] Download GID#4f0c44a50d517e08 not complete: G://G:/test/S2A_OPER_PRD_MSIL1C_PDMC_20160703T230548_R096_V20160703T144853_20160703T144853.SAFE/rep_info/S2_User_Product_Level-1C_Metadata.xsd
Download Results:
gid |stat|avg speed |path/URI
======+====+===========+=======================================================
4f0c44|ERR | 0B/s|G://G:/test/S2A_OPER_PRD_MSIL1C_PDMC_20160703T230548_R096_V20160703T144853_20160703T144853.SAFE/rep_info/S2_User_Product_Level-1C_Metadata.xsd
Status Legend:
(ERR):error occurred.
Note that it doubles up G://G:/ where G is my hard drive letter.
Because it cant download this S2_User_Product_Level-1C_Metadata.xsd file the download process cant resume.
Hope this helps.
Dear Burt, I am sorry, can't help, I do not use windows.
Hope you'll find someone t help you.
Olivier
I got this working by doing the following not sure if it helps anyone.
Removed the Linux if block for 'point' and added double \ just in front of " for start and end of Intersects. And did the same for rectangle. So you only need the following lines. So remove all the lines from 194-204 and add the following and it should work.
if geom=='point':
query_geom='footprint:\"Intersects(%f,%f)\"'%(options.lat,options.lon)
elif geom=='rectangle':
query_geom='footprint:\"Intersects(POLYGON(({lonmin} {latmin}, {lonmax} {latmin}, {lonmax} {latmax}, {lonmin} {latmax},{lonmin} {latmin})))\"'.format(latmin=options.latmin,latmax=options.latmax,lonmin=options.lonmin,lonmax=options.lonmax)
This change is purely to get it working on Windows. If you make this change and run on Linux it will not work. Hope this helps someone.
Note: You need to download aria2 and drop the exe and related files in the folder where python exe is.