sentinelsat.exceptions.ServerError: HTTP status 500: NullPointerException : Object not found
Jan-Oliver opened this issue · comments
Bug/Problem description
Hi there!
I am getting the following error when trying to download certain products. Here, I am attempting to download the product named S2A_MSIL1C_20220507T103631_N0400_R008_T32UMD_20220507T124253 with id c120ac84-9a1a-468f-8bf7-cd787c921203.
Traceback (most recent call last):
File "/home/sentinel/.automation/automation/lib/python3.10/site-packages/sentinelsat/sentinel.py", line 1017, in _check_scihub_response
response.raise_for_status()
File "/home/sentinel/.automation/automation/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: for url: https://scihub.copernicus.eu/dhus/odata/v1/Products('c120ac84-9a1a-468f-8bf7-cd787c921203')/$value
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/sentinel/.automation/automation/lib/python3.10/site-packages/sentinelsat/download.py", line 700, in _download_online_retry
return self.download(uuid, directory, stop_event=stop_event)
File "/home/sentinel/.automation/automation/lib/python3.10/site-packages/sentinelsat/download.py", line 136, in download
filename = self.api._get_filename(product_info)
File "/home/sentinel/.automation/automation/lib/python3.10/site-packages/sentinelsat/sentinel.py", line 596, in _get_filename
self._check_scihub_response(req, test_json=False)
File "/home/sentinel/.automation/automation/lib/python3.10/site-packages/sentinelsat/sentinel.py", line 1063, in _check_scihub_response
raise ServerError(msg, response)
sentinelsat.exceptions.ServerError: HTTP status 500: NullPointerException : Object (c120ac84-9a1a-468f-8bf7-cd787c921203.zip)not found.
ERROR:sentinelsat.SentinelAPI:There was an error downloading S2A_MSIL1C_20220507T103631_N0400_R008_T32UMD_20220507T124253
Traceback (most recent call last):
File "/home/sentinel/.automation/automation/lib/python3.10/site-packages/sentinelsat/sentinel.py", line 1017, in _check_scihub_response
response.raise_for_status()
File "/home/sentinel/.automation/automation/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: for url: https://scihub.copernicus.eu/dhus/odata/v1/Products('c120ac84-9a1a-468f-8bf7-cd787c921203')/$value
As far as I understand, the product is not found server side. This is weird, as I can check if the product is online using the is_online(id) function without any problem. I attached the code I am using so you can hopefully reproduce it.
Code example
from sentinelsat import SentinelAPI
# Create API instance
sentinel_api_instance = SentinelAPI(sentinel_api_username, sentinel_api_password, 'https://scihub.copernicus.eu/dhus/')
# Query parameters
TILE_NAME = "32UMD"
RELATIVE_ORBIT_ID = 8
START_DATE = "20220506"
END_DATE = "20220508"
query_kwargs = {
'platformname': 'Sentinel-2',
'producttype': 'S2MSI1C',
'date': (START_DATE, END_DATE),
'relativeorbitnumber':(RELATIVE_ORBIT_ID)}
query_kwargs['tileid'] = TILE_NAME
query_kwargs['raw'] = f'tileid:{TILE_NAME} OR filename:*_T{TILE_NAME}_*'
# Query available products for parameters
products = sentinel_api_instance.query(**query_kwargs)
print(f"Amount of products found {len(products)}")
# Extract product id and title and print them
product_id = [product_id for product_id in products.keys()][0]
product_title = [products[product_id]['title'] for product_id in products.keys()][0]
print(f"Product title is: {product_title}")
print(f"Product id is: {product_id}")
# Check if the product is online
is_online = sentinel_api_instance.is_online(product_id)
print(f"Is online {is_online}")
# As the product is online, download it
sentinel_api_instance.download_all(products, directory_path=".", max_attempts=1)
The output I am getting is:
Amount of products found 1
Product title is: S2A_MSIL1C_20220507T103631_N0400_R008_T32UMD_20220507T124253
Product id is: c120ac84-9a1a-468f-8bf7-cd787c921203
Is online True
Downloading products: 0%| | 0/1 [00:00<?, ?product/s]ERROR:sentinelsat.SentinelAPI:There was an error downloading S2A_MSIL1C_20220507T103631_N0400_R008_T32UMD_20220507T124253
Traceback (most recent call last):
File "/home/sentinel/.automation/automation/lib/python3.10/site-packages/sentinelsat/sentinel.py", line 1017, in _check_scihub_response
response.raise_for_status()
File "/home/sentinel/.automation/automation/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: for url: https://scihub.copernicus.eu/dhus/odata/v1/Products('c120ac84-9a1a-468f-8bf7-cd787c921203')/$value
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/sentinel/.automation/automation/lib/python3.10/site-packages/sentinelsat/download.py", line 700, in _download_online_retry
return self.download(uuid, directory, stop_event=stop_event)
File "/home/sentinel/.automation/automation/lib/python3.10/site-packages/sentinelsat/download.py", line 136, in download
filename = self.api._get_filename(product_info)
File "/home/sentinel/.automation/automation/lib/python3.10/site-packages/sentinelsat/sentinel.py", line 596, in _get_filename
self._check_scihub_response(req, test_json=False)
File "/home/sentinel/.automation/automation/lib/python3.10/site-packages/sentinelsat/sentinel.py", line 1063, in _check_scihub_response
raise ServerError(msg, response)
sentinelsat.exceptions.ServerError: HTTP status 500: NullPointerException : Object (c120ac84-9a1a-468f-8bf7-cd787c921203.zip)not found.
Downloading products: 100%|██████████| 1/1 [00:00<00:00, 2.65product/s]ERROR:sentinelsat.SentinelAPI:c120ac84-9a1a-468f-8bf7-cd787c921203 failed: Traceback (most recent call last):
File "/home/sentinel/.automation/automation/lib/python3.10/site-packages/sentinelsat/sentinel.py", line 1017, in _check_scihub_response
response.raise_for_status()
File "/home/sentinel/.automation/automation/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: for url: https://scihub.copernicus.eu/dhus/odata/v1/Products('c120ac84-9a1a-468f-8bf7-cd787c921203')/$value
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/sentinel/.automation/automation/lib/python3.10/site-packages/sentinelsat/download.py", line 721, in _download_online_retry
raise last_exception
File "/home/sentinel/.automation/automation/lib/python3.10/site-packages/sentinelsat/download.py", line 700, in _download_online_retry
return self.download(uuid, directory, stop_event=stop_event)
File "/home/sentinel/.automation/automation/lib/python3.10/site-packages/sentinelsat/download.py", line 136, in download
filename = self.api._get_filename(product_info)
File "/home/sentinel/.automation/automation/lib/python3.10/site-packages/sentinelsat/sentinel.py", line 596, in _get_filename
self._check_scihub_response(req, test_json=False)
File "/home/sentinel/.automation/automation/lib/python3.10/site-packages/sentinelsat/sentinel.py", line 1063, in _check_scihub_response
raise ServerError(msg, response)
sentinelsat.exceptions.ServerError: HTTP status 500: NullPointerException : Object (c120ac84-9a1a-468f-8bf7-cd787c921203.zip)not found.
Downloading products: 100%|██████████| 1/1 [00:00<00:00, 2.63product/s]
ResultTuple(downloaded={}, retrieval_triggered={}, failed={'c120ac84-9a1a-468f-8bf7-cd787c921203': {'id': 'c120ac84-9a1a-468f-8bf7-cd787c921203', 'title': 'S2A_MSIL1C_20220507T103631_N0400_R008_T32UMD_20220507T124253', 'size': 768608113, 'md5': '10e8de1b8058878200d5a467af468960', 'date': datetime.datetime(2022, 5, 7, 10, 36, 31, 24000), 'footprint': 'POLYGON((7.500940412097549 53.240200815060746,9.146276872400831 53.24953673463324,9.143007184742661 52.26245957867572,7.534433848467897 52.25344953517325,7.500940412097549 53.240200815060746))', 'url': "https://scihub.copernicus.eu/dhus/odata/v1/Products('c120ac84-9a1a-468f-8bf7-cd787c921203')/$value", 'Online': True, 'Creation Date': datetime.datetime(2022, 5, 7, 16, 13, 2, 145000), 'Ingestion Date': datetime.datetime(2022, 5, 7, 16, 12, 20, 655000), 'quicklook_url': "https://scihub.copernicus.eu/dhus/odata/v1/Products('c120ac84-9a1a-468f-8bf7-cd787c921203')/Products('Quicklook')/$value", 'exception': ServerError('NullPointerException : Object (c120ac84-9a1a-468f-8bf7-cd787c921203.zip)not found.', <Response [500]>)}})
Any help is appreciated!
Best,
Olli
I have been experiencing the same issue with some specific products in the past few weeks. Since I use the node filter function I thought yours was a different issue altogether, but when trying to download the entire zip I got the NullPointer exception too.
For instance in the same date (20220829) the command self.trigger_offline_retrieval(id) in
sentinelsat/sentinelsat/download.py
Line 158 in 7fd4122
But the same command for product S2A_MSIL2A_20220829T101611_N0400_R065_T32TML_20220829T180600 with id c790d363-f9ff-4598-a50d-471420afde1b throws the error "sentinelsat.exceptions.ServerError: HTTP status 500: UnsupportedOperationException : URL stream cannot be opened".
Ignoring the node filter and trying to download the entire product throws the "sentinelsat.exceptions.ServerError: HTTP status 500: NullPointerException : Object (c790d363-f9ff-4598-a50d-471420afde1b.zip)not found".
Weirdly enough, self.api.is_online(id) returns True, so it seems that the API thinks the product is online for some reason, so maybe the bug is indeed on the server side. Do you have any idea what could cause this? Should we just contact Copernicus EO Support and see if they can check on these products?
I contacted Copernicus EO Support at eosupport@copernicus.esa.int and they fixed the issue with the affected products for me, so we can safely say this is indeed a scihub-issue.