Giters

olivierhagolle / Sentinel-download

Automated download of Sentinel-2 L1C data from ESA (through wget) :http://olivierhagolle.github.io/Sentinel-download

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

query_results.xml file is generated, but no Sentinel files are downloaded

kedziorm opened this issue · comments

commented

I'm trying to download subset of the Sentinel-1 data to a specific folder.
It seems that Sentinel-download.py returns some query results, but do not start download.

Is it a bug or should I manually use query_results.xml to download my files?
And why in the messages there's an "401 Unauthorized" if it gets some results?

git clone https://github.com/olivierhagolle/Sentinel-download.git


Cloning into 'Sentinel-download'...
remote: Counting objects: 261, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 261 (delta 0), reused 0 (delta 0), pack-reused 258
Receiving objects: 100% (261/261), 78.58 KiB | 0 bytes/s, done.
Resolving deltas: 100% (129/129), done.
Checking connectivity... done.


cd Sentinel-download/

python Sentinel_download.py --latmin 51.75 --latmax 52.25 --lonmin -23.00 --lonmax 23.75 -a apihub.txt -s S1A*GRDH -w /home/mateuszk/Testy -d 20160508


wget --no-check-certificate --user="guest" --password="guest" --output-document=query_results.xml "https://scihub.copernicus.eu/apihub/search?q=footprint:\"Intersects(POLYGON((-23.0 51.75, 23.75 51.75, 23.75 52.25, -23.0 52.25,-23.0 51.75)))\" filename:S1A*GRDH*&rows=100"
--2016-06-23 13:27:16--  https://scihub.copernicus.eu/apihub/search?q=footprint:%22Intersects(POLYGON((-23.0%2051.75,%2023.75%2051.75,%2023.75%2052.25,%20-23.0%2052.25,-23.0%2051.75)))%22%20filename:S1A*GRDH*&rows=100
Translacja scihub.copernicus.eu (scihub.copernicus.eu)... 131.176.236.28
Łączenie się z scihub.copernicus.eu (scihub.copernicus.eu)|131.176.236.28|:443... połączono.
Żądanie HTTP wysłano, oczekiwanie na odpowiedź... 401 Unauthorized
Wybrano uwierzytelnienie: Basic realm="Sentinels Scientific Data Hub Search"
Łączenie się z scihub.copernicus.eu (scihub.copernicus.eu)|131.176.236.28|:443... połączono.
Żądanie HTTP wysłano, oczekiwanie na odpowiedź... 401 Unauthorized

Uwierzytelnienie użytkownik/hasło nie powiodło się.
Traceback (most recent call last):
  File "Sentinel_download.py", line 228, in <module>
    xml=minidom.parse("query_results.xml")
  File "/usr/lib64/python2.7/xml/dom/minidom.py", line 1918, in parse
    return expatbuilder.parse(file)
  File "/usr/lib64/python2.7/xml/dom/expatbuilder.py", line 924, in parse
    result = builder.parseFile(fp)
  File "/usr/lib64/python2.7/xml/dom/expatbuilder.py", line 211, in parseFile
    parser.Parse("", True)
xml.parsers.expat.ExpatError: no element found: line 1, column 0



echo Now I will edit apihub.txt to provide credentials to my SciHub account
Now I will edit apihub.txt to provide credentials to my SciHub account


gedit apihub.txt


python Sentinel_download.py --downloader aria2 --latmin 51.75 --latmax 52.25 --lonmin -23.00 --lonmax 23.75 -a apihub.txt -s S1A*GRDH -w /home/mateuszk/Testy -d 20160508
wget --no-check-certificate --user="MyAccountName" --password="myPasswordWithSpecialCharactersSuchAs!and@" --output-document=query_results.xml "https://scihub.copernicus.eu/apihub/search?q=footprint:\"Intersects(POLYGON((-23.0 51.75, 23.75 51.75, 23.75 52.25, -23.0 52.25,-23.0 51.75)))\" filename:S1A*GRDH*&rows=100"
--2016-06-23 13:31:35--  https://scihub.copernicus.eu/apihub/search?q=footprint:%22Intersects(POLYGON((-23.0%2051.75,%2023.75%2051.75,%2023.75%2052.25,%20-23.0%2052.25,-23.0%2051.75)))%22%20filename:S1A*GRDH*&rows=100
Translacja scihub.copernicus.eu (scihub.copernicus.eu)... 131.176.236.28
Łączenie się z scihub.copernicus.eu (scihub.copernicus.eu)|131.176.236.28|:443... połączono.
Żądanie HTTP wysłano, oczekiwanie na odpowiedź... 401 Unauthorized
Wybrano uwierzytelnienie: Basic realm="Sentinels Scientific Data Hub Search"
Łączenie się z scihub.copernicus.eu (scihub.copernicus.eu)|131.176.236.28|:443... połączono.
Żądanie HTTP wysłano, oczekiwanie na odpowiedź... 200 OK
Długość: nieznana [application/atom+xml]
Zapis do: `query_results.xml'

query_results.xml       [  <=>               ] 248,48K  1,12MB/s    in 0,2s    

2016-06-23 13:31:38 (1,12 MB/s) - zapisano `query_results.xml' [254447]

77b079c1-084c-4a90-81e5-2aeb36eec0ce
bf219593-4c0c-4c9c-ae07-87050d9665ed
7f826694-07df-4f5c-a046-1a7eccfe86ac
8ae33ebf-8e3c-4e51-9592-8c6df5e76b2d
47de62c1-e8ac-48bb-bc34-c5cea9990670
a3aececd-3837-4f93-89a9-382b8bfe70d8
25a7455e-b4b1-490e-af17-22f4a0a6b835
344a752d-06ad-4a2e-8cb4-554a39bf717f
39f14dc7-357f-48b8-89ef-3f5ae67591fa
61c80884-afe1-40c9-b409-df5cd09fda50
178d8651-3086-42ed-af06-2d2f2afc484f
bfa88079-38e4-4d3d-aa03-2b0015fb7c96
caff0a71-aa3c-4327-a224-b69dbb001842
b66ec94c-9df1-4b0d-b6a4-0bb337352407
151756a1-f5eb-408f-9307-b5dacb7c7a8a
bc0728e2-cca3-4454-8cd7-796de2ee0c86
b6dca0c2-f1dc-427c-9222-3a42e16b0f75
715a0689-8ee7-41d3-95a7-46eac2c6e3d3
b4cea1d6-9fd8-47a3-bb6c-1d281fbb8455
314c5061-faff-430d-8d8b-aff578fc3a79
cbaf9e6b-c105-4419-aeef-d1c253976a1d
438265d4-5e1d-4392-811d-4c96999482eb
9c85b737-1e85-4445-96b4-e6a6bd742593
08bfe4ef-2553-4552-bb63-a895d8f8edc2
28949bdc-53f1-4238-abfa-2b8d0448c5b8
0dd46910-1513-4c96-a000-0bf19113c9d6
0082668c-cf31-4059-86e6-cc18af07c0c6
e33544c1-b6d9-40d0-b74b-b7c7b0f1986a
a1d3d89a-0b4d-4a81-a98f-22e091dfb20b
b8c3dc6b-ff07-4e3a-8da2-5f90842ac65d
035f3e04-86cb-45b8-a9a0-abef1fb4a2e1
60051ab2-0ea0-4ae2-974f-cb157b5bca19
be4304b3-bbc1-4ff4-8c8f-24e610d9e21d
ca1c2bb0-b819-404c-95a8-5743d1eee831
68ba81e6-4e50-4348-9e5a-15cf9006c00c
9b68dbbd-7d49-420a-93cf-3da183a5a830
40c68fc3-a077-499e-a022-c729044246d2
e90ce463-ba24-4547-bde6-f230a53c9a3a
30c5625f-81d3-41b6-8a18-b36dcd108aa0
ef855941-bf32-4116-9e06-6e9267268f1e
2c7eb763-c933-4a6f-b69d-bda09b12c090
370fac88-b4cb-439e-ba7e-da692b7fa272
f8d3185c-5727-4e18-ba80-52ad13177112
51e75ea5-b923-477b-941b-b06a9f6f2b9b
2f21231e-b20e-4469-af59-fde7513d6fda
17758a75-8e54-4c73-8f4a-a09ff40e3306
526a2919-f29b-404d-83ec-197a8c69b782
d23566e7-02a8-4378-a4ec-b32b7d240773
73272b69-69b0-4ecd-b878-0cd462e28a97
13c5be69-728b-496c-910e-e0833424574e
2fc6d817-a933-441d-a292-b120b85ffd95
932fb469-fdd8-4f39-8210-2046d7e5a527
ce362374-36b4-41eb-8774-9a5cefc5f1d0
da74214b-30ad-4425-90d9-a10454113e7b
d171e19b-64ef-46e6-bcbf-6d3ff873eba1
8c10351c-9305-46e1-aab9-90591bd851ae
d4213bd0-206c-4b64-9af7-494c670e5220
ba23bb88-9c4a-423d-8e0e-3ff05b9b4c40
97b04321-43af-4115-a666-c36a92759582
7383b950-50ef-4def-8de6-73d0439e44f7
e560973a-dbb5-4d3c-a17a-9bfd51814ace
ea450601-edc7-44c4-84b1-26ca14d324b5
9ed8cda5-5e8b-4167-9eb0-77b2caa374ff
b1604df3-e6e7-4ca4-ba9c-8acc1889eae8
f1487f2e-a7b2-45b4-ad72-5b4546ffc40f
d7247646-08af-4b72-8eb4-8cbaedce606c
58537076-82e4-48e9-b1f1-b1cf59dd8eee
30650919-6a99-4c86-bf9a-d4fab71add14
e79c7174-e17f-4062-8d6d-7ed3060a01b7
1cafdd7f-f872-43a5-8a4d-0e64c939ca76
9d7b3327-421e-4494-98db-34c061e5fb5a
72a82e9e-1464-490e-9ccf-aeddfc702b56
f58464ce-2d90-4002-8a11-b20498569be8
c505b853-ea57-4f0d-8168-1ca70978426e
cac4f1b4-f166-46d0-80f5-a93567051ca6
8cd6632a-926d-4732-a0da-a603d99e3ccb
47467643-54ea-42a9-b208-ff02dbea509b
e9d91fc0-46cd-4686-b3e8-e2907cad6a6e
7faa8cab-5f22-4326-b8b1-e0f395ead117
e9cc9aa1-9f5f-46df-8a3c-96e548d2dd31
02395fe6-8839-4225-a297-e2ce5801d1a1
2f7ff185-51b6-4b01-9478-298ecbee4a72
29d26e21-a53e-4483-999b-222306dd8609
da10ef2a-7de9-43ae-a449-c7f00a1d9cc2
b574fb96-70d7-419a-983a-43324eccd896
995afd03-6614-4172-8cac-8abd0b60eeff
f5b45ab6-d384-4d2b-a7c4-53d329762a7a
85257b7a-5969-43fc-a034-79421ba2b86e
6bb7ea28-591b-4c90-be76-f08a47f1adfb
e87edac9-5adf-437f-880d-b5d0fc61d8d6
37ff0cc9-462e-4d97-9f13-cf824050d0f2
3af6197f-67d1-4d63-91c0-685570af40cf
f0b9828e-5fcb-4ada-86de-1f217b98339e
d50796b4-11b9-4cd5-b998-f47f215df634
eebb913c-b2a0-4a87-8025-5fa87e7fd0db
49742d08-9227-465f-808d-531ba9f75c32
415ed0c9-f9f0-4076-afc0-c3af2605e5fb
c6102ed7-db02-46de-92ac-1a8576ec07cf
b4264397-44db-4371-86a9-74949e5c35c7
66cac5c8-1bcf-4f95-b469-6f9bbbe10593


ls /home/mateuszk/Testy/
commented

I tried the following command, and it started downloading some products.
python Sentinel_download.py --latmin 51.75 --latmax 52.25 --lonmin -23.00 --lonmax 23.75 -a apihub.txt -s S1A*GRDH -d 20160508

So sorry, not able to reproduce your issue
Olivier

commented

Dear @olivierhagolle

but as I can see you're not providing -w parameter to save data in different path (in my case -w /home/mateuszk/Testy)

commented

You may try without -w too.
Olviier

commented

Okay, I checked.
Without -w option, I was able to download files (I use Python 2.7.11+, Ubuntu 16.04 LTS)

Previously, I've tried to download files to the folder on the external hard drive.
This time, I've tried to use -w option providing path to the folder on local partition. It seems that it works properly!

Maybe this is caused by the lack of permission to save?
I noticed that scripts do not verify if there's enough space to save file. If unable to save, script says about "success" and tries to download next file.

I've also noticed that although we provide latitude and longitude, downloaded files are not cropped to that area. Is it correct/is it the way it was designed?

commented

Dear Kedziorm,
yes, the tool just downloads the products that intersect the box or contain the point. it does not crop the files.

Thanks for the remark about the check of success. I should try to control the file size, but for that, I have to get it before the download. I'll have to search how to do that. There must be an option for it in wget.

Best regards,
Olivier

commented

Dear Olivier,

I'm trying to download sentinel 2 data but it seems that the file "query_results.xml" is not generated.
Would you know where could this issue come from ?
Thank you very much,

Best regards, Amelie

commented

Dear Amelie,
you should have opened a new issues, as this one is closed, I did not see it. And besides, i was on holidays ;).
I lack information to check, could you please download again the tool (as some changes with made), and provide me the command line you used (opening a new issue).
Cordialement,
Olivier

commented

Dear Olivier,
I'm trying to download sentinel 2 data, however the file "query_results.xml" is empty. The command and result are as follow:

python Sentinel_download.py --lat 43.6 --lon 1.44 -a apihub.txt -o 51 -d 20151226 -s S2

wget --no-check-certificate --user="guest" --password="guest" --output-document=query_results.xml "https://scihub.copernicus.eu/apihub/search?q=footprint:"Intersects(43.600000,1.440000)" filename:S2R051 producttype:S2MSI1C &rows=100"
--2017-06-12 22:43:22-- https://scihub.copernicus.eu/apihub/search?q=footprint:Intersects(43.600000,1.440000)%20filename:S2*R051*%20producttype:S2MSI1C%20&rows=100
Resolving scihub.copernicus.eu (scihub.copernicus.eu)... 131.176.236.28
Connecting to scihub.copernicus.eu (scihub.copernicus.eu)|131.176.236.28|:443... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Authentication selected: Basic realm="Sentinels Scientific Data Hub Search"
Reusing existing connection to scihub.copernicus.eu:443.
HTTP request sent, awaiting response... 401 Unauthorized

Username/Password Authentication Failed.
Traceback (most recent call last):
File "Sentinel_download.py", line 280, in
xml=minidom.parse("query_results.xml")
File "C:\CONDA2.7\lib\xml\dom\minidom.py", line 1918, in parse
return expatbuilder.parse(file)
File "C:\CONDA2.7\lib\xml\dom\expatbuilder.py", line 924, in parse
result = builder.parseFile(fp)
File "C:\CONDA2.7\lib\xml\dom\expatbuilder.py", line 211, in parseFile
parser.Parse("", True)
xml.parsers.expat.ExpatError: no element found: line 1, column 0

Then I edited apihub.txt to provide credentials to my SciHub account (https://scihub.copernicus.eu/dhus/#/home). However the error still exists.

Can you give me some advice? Thank you.
Best regards,
Liangzhong

commented

You should introduce your scihub account and password in apihub.txt file.
Best regards,
Olivier