Is there a way to get the entire catalog of GUNW of metadata?
cmarshak opened this issue · comments
Is your feature request related to a problem? Please describe.
I want to get all the metadata for the entire catalog of GUNWs and make a pretty picture. I always get a subset (~150k products when there is 400k+). Do I have split up the globe?
Describe the solution you'd like
results = asf.search(processingLevel=[asf.PRODUCT_TYPE.GUNW_STD], maxResults=1_000_000) gives me all GUNW metadata
Describe alternatives you've considered
Not much.
Additional context
Make pretty pictures.
I see similar truncated search results with asf_search 3.2.1 on python 3.10.4:
>>> import asf_search
>>> results = asf_search.search(processingLevel=[asf_search.PRODUCT_TYPE.GUNW_STD])
>>> len(results)
112831
>>> asf_search.__version__
'3.2.1'
I'd expect over 471,000 results as of this writing:
https://search.asf.alaska.edu/#/?dataset=SENTINEL-1%20INTERFEROGRAM%20(BETA)&productTypes=GUNW_STD
https://api.daac.asf.alaska.edu/services/search/param?processingLevel=GUNW_STD&output=count
https://cmr.earthdata.nasa.gov/search/granules?provider=ASF&collection_concept_id=C1595422627-ASF&page_size=0
Currently, asf_search relies on our SearchAPI as its backend, which has a ~15-minute time limit on how long it'll allow queries to run, it's possible you're running into that. We have some backlog to alleviate that, but in the mean time, a simple solution would be to break up the query by years (probably simpler than by geospatial).
Some quick checking indicates each year there are fewer than 100k products (about 90k in 2018 and 2019 each, significantly less in other years), which should complete well within the SearchAPI's time limit.
Thank you!