[BUG] ariaExtract: multi-looking results in all nans in the geometry files
jlmaurer opened this issue · comments
Describe the bug
Running ariaExtract.py with -ml option set to 2 results in all invalid pixels in the geometry files (incidenceAngle, lookAngle, azimuthAngle, bParallel, bPerpendicular)
To Reproduce
Steps to reproduce the behavior:
- Download a set of ARIA interferograms and put them in products/
- Create an 'output' directory to store products
- Command used: ariaExtract.py -f 'products/S1-GUNW*.nc' -w output -l 'all' -d Download -ml 2
- Output files listed above do not contain any valid pixels, and the DEM contains all zeros.
Running the same command without the -ml argument works fine.
Expected behavior
Get valid output geometry files that correspond to the multilooked IFGs
Desktop (please complete the following information):
- ARIA-tools git tag: [e.g. git show --summary]: 0bff9fc
- OS: Mac OS Big Sur 11.2.3 (20D91)
@jlmaurer following up the discussion we had in the meeting today, an easy test to see if the issue is the overlap is to just run without multi-looking and see if the invalid pixels are still present.
Hi @bbuzz31 thanks for that! I checked both versions (no multi-looking and with multi-looking). The image shows that the full-res file seems to work just fine (with of course invalid pixels outside the bounds of the frame). Not sure whether this makes sense based on what you were saying?
Command for hi-res version:
ariaExtract.py -f 'products/S1-GUNW-A-R-077-tops-2020*.nc' -w output_hires -l 'all' -d Download
Command for lo-res version:
ariaExtract.py -f 'products/S1-GUNW-A-R-077-tops-2020*.nc' -w output -l 'all' -d Download -ml 2
The images are the interferograms produced (or not), just opening them in python and plotting.
Thanks for sharing. I think this means that the minimum overlap is NOT the problem, perhaps @sssangha can comment.
The next easiest place to check for bug would probably be the masking. The value 0 was recently introduced to represent nans following MintPy, which I noticed caused a bug in tsPlot
@sssangha that's great! What was causing the issue?
@jlmaurer I should clarify that while I wasn't able to replicate your exact issue. There were problems with attempting to download and manage large DEMs because the product shapefiles were being leveraged as opposed to the actual broader, grid extent of the region. It is possible this may translate to empty fields downstream, but I wasn't able to replicate it after my fixes.
@jlmaurer we will cut a release for ARIA-tools, but want to make sure this is not an issue. Could you give it another test when you are back from holiday?
@jlmaurer Brett appears to have replicated your issue with an older version of gdal (3.1.4), so I did a completely new install where I rebuilt the environment from scratch (ended up with goal 3.2.1), and find that this issue does not arise. See DEM and incidence angle below:
I'm thinking that this may not be just a problem associated with an older version of gdal per se, but perhaps a combination of packages which result in this issue. I would suggest to completely rebuilt a new conda environment from scratch as I did and hopefully you can confirm my findings. Would you or @bbuzz31 have some bandwidth for this? @rzinke did you recently encounter such an issue?
Many thanks!
@sssangha that is a good point, I will try rebuilding the environment from scratch and see if that works.
@sssangha I rebuilt from scratch with gdal v3.1.2.
I'm now encountering an error message.
Also, I've noticed you aren't running on the same track. May be better to focus on one before testing on others.
ariaDownload.py -t 077-s 20200101 -e 20200201
ariaExtract.py -f 'products/S1-GUNW-A-R-077-tops-2020*.nc' -l 'all' -d Download -ml 2
Errors:
ARIA-tools Version: 1.1.0
*****************************************************************
*** Extract Product Function ***
*****************************************************************
Multi-core version
All (8) GUNW products meet spatial bbox criteria.
Group GUNW products into spatiotemporally continuous interferograms.
All (1) interferograms are spatially continuous.
All layers are to be extracted, pass all keys.
Thread count specified for gdal multiprocessing = 2
WARNING: User-defined bounds results in an area of 684150 km which exceeds the maximum download area of 400000; downloading in chunks
^[Warning 1: the input vector layer has a SRS, but the source raster dataset does not.
Cutline results may be incorrect.
Applied cutline to produce 3 arc-sec SRTM DEM: ./DEM/SRTM_3arcsec.dem
Generating: unwrappedPhase - [==================================================] 20200124_20200112NETCDF:"/Volumes/BB_4TB/data/VLM/Sentinel1/track_077/products/S1-GUNW-A-R-077-tops-20200124_
20200112-231342-33720N_31670N-PP-3e2b-v2_0_2.nc":/science/grids/data/unwrappedPhase is GDAL compatible
NETCDF:"/Volumes/BB_4TB/data/VLM/Sentinel1/track_077/products/S1-GUNW-A-R-077-tops-20200124_20200112-231342-33720N_31670N-PP-3e2b-v2_0_2.nc":/science/grids/data/connectedComponents is GDAL compatible
(many more of these omitted for brevity)
Traceback (most recent call last):
File "/Users/bb/Miniconda3/envs/ARIA-tools-test/bin/ariaExtract.py", line 21, in <module>
main(inps)
File "/Users/bb/Miniconda3/envs/ARIA/lib/python3.8/site-packages/ARIAtools/extractProduct.py", line 1193, in main
export_products(standardproduct_info.products[1], standardproduct_info.bbox_file,
File "/Users/bb/Miniconda3/envs/ARIA/lib/python3.8/site-packages/ARIAtools/extractProduct.py", line 657, in export_products
product_stitch_overlap(unw_files,conn_files,prod_bbox_files,bounds,prods_TOTbbox, outFileUnw=outFileUnw,outFileConnComp= outFileConnComp, mask=mask,outputFormat = outputFormat,verbose=verbose)
File "/Users/bb/Miniconda3/envs/ARIA/lib/python3.8/site-packages/ARIAtools/unwrapStitching.py", line 1458, in product_stitch_overlap
unw.UnwrapOverlap()
File "/Users/bb/Miniconda3/envs/ARIA/lib/python3.8/site-packages/ARIAtools/unwrapStitching.py", line 340, in UnwrapOverlap
self.__calculateCyclesOverlap__()
File "/Users/bb/Miniconda3/envs/ARIA/lib/python3.8/site-packages/ARIAtools/unwrapStitching.py", line 387, in __calculateCyclesOverlap__
connCompFile1 = gdal.Warp('', self.ccFile[counter], options=gdal.WarpOptions(format="MEM", cutlineDSName=outname, outputBounds=polyOverlap.bounds, dstNodata=connCompNoData1))
File "/Users/bb/Miniconda3/envs/ARIA/lib/python3.8/site-packages/osgeo/gdal.py", line 677, in Warp
return wrapper_GDALWarpDestName(destNameOrDestDS, srcDSTab, opts, callback, callback_data)
File "/Users/bb/Miniconda3/envs/ARIA/lib/python3.8/site-packages/osgeo/gdal.py", line 4319, in wrapper_GDALWarpDestName
return _gdal.wrapper_GDALWarpDestName(*args)
RuntimeError: Cannot find coordinate operations from `GEOGCRS["unknown",DATUM["unnamed",ELLIPSOID["Spheroid",6378137,298.257223563,LENGTHUNIT["metre",1,ID["EPSG",9001]]]],PRIMEM["Greenwich",0,ANGLEUNIT["degree",0.0174532925199433,ID["EPSG",9122]]],CS[ellipsoidal,2],AXIS["latitude",north,ORDER[1],ANGLEUNIT["degree",0.0174532925199433,ID["EPSG",9122]]],AXIS["longitude",east,ORDER[2],ANGLEUNIT["degree",0.0174532925199433,ID["EPSG",9122]]]]' to `GEOGCRS["WGS 84",DATUM["World Geodetic System 1984",ELLIPSOID["WGS 84",6378137,298.257223563,LENGTHUNIT["metre",1]]],PRIMEM["Greenwich",0,ANGLEUNIT["degree",0.0174532925199433]],CS[ellipsoidal,2],AXIS["latitude",north,ORDER[1],ANGLEUNIT["degree",0.0174532925199433]],AXIS["longitude",east,ORDER[2],ANGLEUNIT["degree",0.0174532925199433]],ID["EPSG",4326]]'
Note the black squares in this UNCROPPED DEM and the weird black line through the middle. Is something not getting stitched correctly?
@bbuzz31 I tried it again using your track and commands, and didn't encounter the same issue nor found a gap in my DEM. Below are the DEM and azimuth angle field:
It is possible that perhaps our environments are slightly different.
@bbuzz31 as a follow-up, I think it would be best for me to share a record of this environment here that you can use directly.