[BUG] DEM download fails for large areas

Question

[BUG] DEM download fails for large areas

bbuzz31 opened this issue 3 years ago · comments

Describe the bug
ARIA area calculation for DEM extent to decide whether to chunk or not does not match the SRTM servers.
Relevant line is 349 in extractProduct.py

Note the area calculation (shapefile_area) is correct.

To Reproduce
Steps to reproduce the behavior:

ariaDownload.py -t 004 -s 20191201 -e 20200101 # 13 products
ariaExtract.py -f "./products/*.nc" -l all -w . -d Download

  File "/Users/bb/Miniconda3/envs/ARIA/bin/ariaExtract.py", line 16, in <module>
    main(inps)
  File "/Users/bb/Miniconda3/envs/ARIA/lib/python3.8/site-packages/ARIAtools/extractProduct.py", line 942, in main
    inps.demfile, demfile, Latitude, Longitude = prep_dem(inps.demfile,
  File "/Users/bb/Miniconda3/envs/ARIA/lib/python3.8/site-packages/ARIAtools/extractProduct.py", line 310, in prep_dem
    gdal.Warp(aria_dem, demfilename, format=outputFormat,
  File "/Users/bb/Miniconda3/envs/ARIA/lib/python3.8/site-packages/osgeo/gdal.py", line 665, in Warp
    srcDSTab = [Open(srcDSOrSrcDSTab)]
  File "/Users/bb/Miniconda3/envs/ARIA/lib/python3.8/site-packages/osgeo/gdal.py", line 4192, in Open
    return _gdal.Open(*args)
RuntimeError: `./DEM/SRTM_3arcsec_uncropped.tif' not recognized as a supported file format.

We calculate the area of ./productBoundingBox/productBoundingBox_croptounion_formetadatalyr.json as 84497 km^2, which is correct, but ./DEM/SRTM_3arcsec_uncropped.tif reads:
Error: The maximum area for SRTMGL1_E is 450,000 km2. The selected area is 472,979 km2

BB · Answer 1 · Sun May 23 2021 02:41:55 GMT+0800 (China Standard Time)

In #269 I put in a hack to initialize downloading in chunks if the area we calculate is greater than 80000 km^2, which solves the problem (for now at least).

rzinke · Answer 2 · Tue Jun 29 2021 02:03:12 GMT+0800 (China Standard Time)

Hi @bbuzz31, I am still getting an error from this after updating to the most recent version of AT and reinstalling.

$ git log
> commit 0bff9fc6568e7856606fad82e74547f8cfcc623b (HEAD -> dev, origin/dev, origin/HEAD)
Author: BB <bbuzz318@icloud.com>
Date:   Tue May 25 06:27:48 2021 -0700

Oddly, this area is smaller than some of the ones I had been running previously.

Here is my launch command:

$ ariaTSsetup.py -f "Nproducts/*.nc" -w ML3 -b /AOIs/114_A_Tibet_AOI_croppedN.shp -d download -m download -nt 8 -ml 3

And here is the error:

et/Nproducts/S1-GUNW-A-R-114-tops-20200526_20191116-121001-43771N_41735N-PP-b5f6-v2_0_3.nc'
Thread count specified for gdal multiprocessing = 8
Download/cropping DEM
Traceback (most recent call last):
  File "/u/sar-r0/rzinke/python/miniconda3/envs/ARIA-tools/bin/ariaTSsetup.py", line 15, in <module>
    main(inps)
  File "/u/sar-r0/rzinke/python/miniconda3/envs/ARIA-tools/lib/python3.8/site-packages/ARIAtools/tsSetup.py", line 420, in main
    inps.demfile, demfile, Latitude, Longitude = prep_dem(
  File "/u/sar-r0/rzinke/python/miniconda3/envs/ARIA-tools/lib/python3.8/site-packages/ARIAtools/extractProduct.py", line 355, in prep_dem
    gdal.Warp(aria_dem, demfilename, format=outputFormat,
  File "/u/sar-r0/rzinke/python/miniconda3/envs/ARIA-tools/lib/python3.8/site-packages/osgeo/gdal.py", line 613, in Warp
    srcDSTab = [Open(srcDSOrSrcDSTab)]
  File "/u/sar-r0/rzinke/python/miniconda3/envs/ARIA-tools/lib/python3.8/site-packages/osgeo/gdal.py", line 3308, in Open
    return _gdal.Open(*args)
RuntimeError: `ML3/DEM/SRTM_3arcsec_uncropped.tif' not recognized as a supported file format.

BB · Answer 3 · Tue Jun 29 2021 02:16:53 GMT+0800 (China Standard Time)

Hey @rzinke. This has something to do with files and/or variables being incorrectly re-used. Haven’t tracked it down yet, but it should work if you start from a clean directory. Although, this may be overkill; pretty sure it will work if you run again after removing just the `productBoundingBox` directory.

…

On Jun 28, 2021, at 11:03 AM, rzinke ***@***.***> wrote: Hi @bbuzz31 <https://github.com/bbuzz31>, I am still getting an error from this after updating to the most recent version of AT and reinstalling. $ git log > commit 0bff9fc (HEAD -> dev, origin/dev, origin/HEAD) Author: BB ***@***.***> Date: Tue May 25 06:27:48 2021 -0700 Oddly, this area is smaller than some of the ones I had been running previously. Here is my launch command: $ ariaTSsetup.py -f "Nproducts/*.nc" -w ML3 -b /AOIs/114_A_Tibet_AOI_croppedN.shp -d download -m download -nt 8 -ml 3 And here is the error: et/Nproducts/S1-GUNW-A-R-114-tops-20200526_20191116-121001-43771N_41735N-PP-b5f6-v2_0_3.nc' Thread count specified for gdal multiprocessing = 8 Download/cropping DEM Traceback (most recent call last): File "/u/sar-r0/rzinke/python/miniconda3/envs/ARIA-tools/bin/ariaTSsetup.py", line 15, in <module> main(inps) File "/u/sar-r0/rzinke/python/miniconda3/envs/ARIA-tools/lib/python3.8/site-packages/ARIAtools/tsSetup.py", line 420, in main inps.demfile, demfile, Latitude, Longitude = prep_dem( File "/u/sar-r0/rzinke/python/miniconda3/envs/ARIA-tools/lib/python3.8/site-packages/ARIAtools/extractProduct.py", line 355, in prep_dem gdal.Warp(aria_dem, demfilename, format=outputFormat, File "/u/sar-r0/rzinke/python/miniconda3/envs/ARIA-tools/lib/python3.8/site-packages/osgeo/gdal.py", line 613, in Warp srcDSTab = [Open(srcDSOrSrcDSTab)] File "/u/sar-r0/rzinke/python/miniconda3/envs/ARIA-tools/lib/python3.8/site-packages/osgeo/gdal.py", line 3308, in Open return _gdal.Open(*args) RuntimeError: `ML3/DEM/SRTM_3arcsec_uncropped.tif' not recognized as a supported file format. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#265 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADWNVR6JT6HD73UBX4WMXWLTVC2OXANCNFSM45HQNG7A>.

rzinke · Answer 4 · Tue Jun 29 2021 03:12:50 GMT+0800 (China Standard Time)

I removed the productBoundingBox and reran, but that didn't work. I get the same error.

BB · Answer 5 · Tue Jun 29 2021 03:21:35 GMT+0800 (China Standard Time)

Doesn’t look like you’re on the latest branch:

…

On Jun 28, 2021, at 12:12 PM, rzinke ***@***.***> wrote: I removed the productBoundingBox and reran, but that didn't work. I get the same error. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#265 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADWNVR3OJCHDSQPKAEES27DTVDCTXANCNFSM45HQNG7A>.

rzinke · Answer 6 · Tue Jun 29 2021 04:37:35 GMT+0800 (China Standard Time)

I updated to the latest branch and reran setup.py:

commit a79e8a1f320724bb3fd63fa4a0e5a537034b55b2 (HEAD -> dev, origin/dev, origin/HEAD)
Author: BB <bbuzz318@icloud.com>
Date:   Sat Jun 26 08:05:43 2021 -0700

I then cleared the old work directory.

Same error message.

BB · Answer 7 · Tue Jun 29 2021 23:12:25 GMT+0800 (China Standard Time)

Can you provide a MWE?

I.e., an ariaDownload command with a subset of the products to recreate the error and as few flags as possible?
Does the bug still occur without masking and/or multilooking? On a single thread?

Also, what is contained in the erroneous file?

dbekaert · Answer 8 · Tue Jul 13 2021 23:54:28 GMT+0800 (China Standard Time)

Also when reporting the minimum working example, you would need to add in the information like the shapefile etc.

@rzinke:

can you confirm if this an issue on ARIA-tools or on Open Topo
Can you clarify if this is transient, gets resolved upon re-run.
If so, perhaps could you check if the open topo API comes with a check if tiles are correctly downloaded?

rzinke · Answer 9 · Wed Jul 14 2021 00:02:36 GMT+0800 (China Standard Time)

I can take a look next week, I'm out of the office at the moment.
Still an issue for me and happy to help debug.

Simran S Sangha · Answer 10 · Tue Jul 20 2021 23:54:16 GMT+0800 (China Standard Time)

Hello @rzinke , my latest PR #275 adjusts chunking for DEMs. Please let me know if this helps.

rzinke · Answer 11 · Thu Jul 22 2021 23:40:20 GMT+0800 (China Standard Time)

That appears to have fixed the issue. Will update you if I find out otherwise.
Cheers.

dbekaert · Answer 12 · Fri Jul 30 2021 05:52:16 GMT+0800 (China Standard Time)

Thanks for confirming @rzinke

rzinke · Answer 13 · Fri Jan 21 2022 04:13:51 GMT+0800 (China Standard Time)

Hello, I am still getting this error.

ariaDownload.py -w Products/Tr004 -t 004 -b "36.75 37.225 -76.655 -75.928" -s 20180101 -e 20200118

ariaExtract.py -f "Products/Tr004/*.nc" -w FullExtract -d download -m download -verbose

gives the error:

ARIA-tools Version: 1.1.0
*****************************************************************
*** Extract Product Function ***
*****************************************************************
Multi-core version
All (189) GUNW products meet spatial bbox criteria.
Group GUNW products into spatiotemporally continuous interferograms.
All (189) interferograms are spatially continuous.
No layers specified; only creating bounding box shapes
Thread count specified for gdal multiprocessing = 2
***Downloading water mask... ***
Traceback (most recent call last):
  File "/Users/rzinke/opt/miniconda3/envs/ARIA-tools/bin/ariaExtract.py", line 21, in <module>
    main(inps)
  File "/Users/rzinke/opt/miniconda3/envs/ARIA-tools/lib/python3.10/site-packages/ARIAtools/extractProduct.py", line 1185, in main
    inps.demfile, demfile, Latitude, Longitude = prep_dem(inps.demfile,
  File "/Users/rzinke/opt/miniconda3/envs/ARIA-tools/lib/python3.10/site-packages/ARIAtools/extractProduct.py", line 355, in prep_dem
    gdal.Warp(aria_dem, demfilename, format=outputFormat,
  File "/Users/rzinke/opt/miniconda3/envs/ARIA-tools/lib/python3.10/site-packages/GDAL-3.4.1-py3.10-macosx-10.9-x86_64.egg/osgeo/gdal.py", line 695, in Warp
    srcDSTab = [Open(srcDSOrSrcDSTab)]
  File "/Users/rzinke/opt/miniconda3/envs/ARIA-tools/lib/python3.10/site-packages/GDAL-3.4.1-py3.10-macosx-10.9-x86_64.egg/osgeo/gdal.py", line 4576, in Open
    return _gdal.Open(*args)
RuntimeError: `FullExtract/DEM/SRTM_3arcsec_uncropped.tif' not recognized as a supported file format.

real	5m34.715s
user	2m12.745s
sys	0m13.864s

I have updated to the latest version:

$ git show --summary
> commit 0251a1dce5230378a2391e54467343e662fdfadf (HEAD -> cutline_dstalpha, origin/dev, origin/HEAD, aria-tools/dev, dev)
Author: BB <bbuzz318@icloud.com>
Date:   Mon Nov 29 11:05:23 2021 -0800

    bug fix on subsetting (#286)

BB · Answer 14 · Fri Jan 21 2022 06:06:39 GMT+0800 (China Standard Time)

I think this is something to do with openTopo.

The tif file says we need an API key

vi ./DEM/SRTM_3arcsec_0_uncropped.tif:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><error>Error: This dataset requires an API Key for access.</error>

Simran S Sangha · Answer 15 · Fri Jan 21 2022 06:22:09 GMT+0800 (China Standard Time)

@rzinke @bbuzz31 yeah, I am also getting this error with an independent test run that worked just a couple days ago or so... We will have to investigate this/potentially tweak fetching from openTopo again