[BUG] Bbox error when filtering by minimum overlap

Question

[BUG] Bbox error when filtering by minimum overlap

rzinke opened this issue 2 years ago · comments

Describe the bug
To plot the spatial extent of my ARIA products, I am attempting to filter out scenes using the "minimum overlap" threshold (-mo option in ariaPlot.py). Doing so causes the error described below, indicating that the productBoundingBox for one or more pairs of dates returns "NoneType". It is worth noting that when I use the -croptounion option, or set the -mo to a smaller value, this error does not appear.

To Reproduce
Steps to reproduce the behavior:

Download

ariaDownload.py -t 48 -b 23.0 55.0 63.0 117.0 -s 20160101 -e 20170101 -nt 4 -v

Command used
ariaPlot.py -f products/*.nc -b ./AOI/track_048_TibetDii.shp -plottracks -nt 4 --figwidth wide -mo 250000
Error Output
Traceback (most recent call last):
File "/opt/miniconda3/envs/ARIA-tools/bin/ariaPlot.py", line 21, in
main(inps)
File "/opt/miniconda3/envs/ARIA-tools/lib/python3.10/site-packages/ARIAtools/productPlot.py", line 616, in main
prods_TOTbbox_metadatalyr, arrshape, proj = merged_productbbox(
File "/opt/miniconda3/envs/ARIA-tools/lib/python3.10/site-packages/ARIAtools/extractProduct.py", line 482, in merged_productbbox
prods_bbox=open_shapefile(scene['productBoundingBox'][0], 0, 0)
File "/opt/miniconda3/envs/ARIA-tools/lib/python3.10/site-packages/ARIAtools/shapefile_util.py", line 34, in open_shapefile
file_bbox = file_bbox.GetLayerByIndex(lyrind).GetFeature(ftind)
AttributeError: 'NoneType' object has no attribute 'GetLayerByIndex'

Expected behavior
I would expect the track latitude extents to be plotted, excluding the scenes that overlap the AOI by less than 250,000 km2.

Desktop (please complete the following information):

ARIA-tools git tag: commit f7a6340 (HEAD -> dev, ariatools/dev, reject_by_extent)
OS: MacOS Big Sur v 11.6.6

Additional context
Might be helpful to see the products being output. To show this, I inserted print('Scene: {:s}'.format(scene['productBoundingBox'][0])) in Line 493 of extractProduct.py, and some print statements to that same for loop in merged_productbbox.

Scene: ./productBoundingBox/20160424_20160331.json
	area: 29065 - rejected
Scene: ./productBoundingBox/20160425_20160401.json
	area: 248618 - rejected
Scene: ./productBoundingBox/20160519_20160424.json
	area: 276831 - OK
Scene: ./productBoundingBox/20160612_20160331.json
	area: 276834 - OK
Scene: ./productBoundingBox/20160612_20160519.json
	area: 276834 - OK
Scene: ./productBoundingBox/20160706_20160401.json
	area: 85656 - rejected
Scene: ./productBoundingBox/20160706_20160424.json
	area: 114020 - rejected
Scene: ./productBoundingBox/20160706_20160612.json
	area: 114020 - rejected
Scene: ./productBoundingBox/20160730_20160425.json
	area: 248392 - rejected
Scene: ./productBoundingBox/20160730_20160519.json
	area: 276610 - OK
Scene: ./productBoundingBox/20160823_20160519.json
	area: 276610 - OK
Scene: ./productBoundingBox/20160823_20160612.json
	area: 276610 - OK
Scene: ./productBoundingBox/20160823_20160706.json
	area: 113892 - rejected
Scene: ./productBoundingBox/20160823_20160730.json
	area: 276610 - OK
Scene: ./productBoundingBox/20160916_20160331.json
	area: 276610 - OK
Scene: ./productBoundingBox/20160916_20160519.json
	area: 276610 - OK
Scene: ./productBoundingBox/20160916_20160612.json
	area: 276610 - OK
Scene: ./productBoundingBox/20160916_20160706.json
	area: 113925 - rejected
Scene: ./productBoundingBox/20160916_20160823.json
	area: 276610 - OK
Scene: ./productBoundingBox/20161015_20160401.json
	area: 43192 - rejected
Scene: ./productBoundingBox/20161015_20160424.json
	area: 29053 - rejected
Scene: ./productBoundingBox/20161015_20160425.json
	area: 43192 - rejected
Scene: ./productBoundingBox/20161015_20160706.json
	area: 71566 - rejected
Scene: ./productBoundingBox/20161015_20160706.json
Traceback (most recent call last):
  File "/opt/miniconda3/envs/ARIA-tools/bin/ariaPlot.py", line 21, in <module>
    main(inps)
  File "/opt/miniconda3/envs/ARIA-tools/lib/python3.10/site-packages/ARIAtools/productPlot.py", line 616, in main
    prods_TOTbbox_metadatalyr, arrshape, proj = merged_productbbox(
  File "/opt/miniconda3/envs/ARIA-tools/lib/python3.10/site-packages/ARIAtools/extractProduct.py", line 494, in merged_productbbox
    prods_bbox=open_shapefile(scene['productBoundingBox'][0], 0, 0)
  File "/opt/miniconda3/envs/ARIA-tools/lib/python3.10/site-packages/ARIAtools/shapefile_util.py", line 34, in open_shapefile
    file_bbox = file_bbox.GetLayerByIndex(lyrind).GetFeature(ftind)
AttributeError: 'NoneType' object has no attribute 'GetLayerByIndex'

It's odd that the pair 20161015_20160706 appears twice.

rzinke · Answer 1 · Thu Jun 23 2022 06:56:15 GMT+0800 (China Standard Time)

Track048_Tibet_AOI.zip

Here is the AOI shapefile that I used.

rzinke · Answer 2 · Thu Jun 23 2022 07:32:04 GMT+0800 (China Standard Time)

It appears that 20161015_20160706 and several other date pairs appear twice in the product_dict, i.e.,

Product dict
./productBoundingBox/20160424_20160331.json
./productBoundingBox/20160425_20160401.json
./productBoundingBox/20160519_20160424.json
./productBoundingBox/20160612_20160331.json
./productBoundingBox/20160612_20160519.json
./productBoundingBox/20160706_20160401.json
./productBoundingBox/20160706_20160424.json
./productBoundingBox/20160706_20160612.json
./productBoundingBox/20160730_20160425.json
./productBoundingBox/20160730_20160519.json
./productBoundingBox/20160823_20160519.json
./productBoundingBox/20160823_20160612.json
./productBoundingBox/20160823_20160706.json
./productBoundingBox/20160823_20160730.json
./productBoundingBox/20160916_20160331.json
./productBoundingBox/20160916_20160519.json
./productBoundingBox/20160916_20160612.json
./productBoundingBox/20160916_20160706.json
./productBoundingBox/20160916_20160823.json
./productBoundingBox/20161015_20160401.json
./productBoundingBox/20161015_20160424.json
./productBoundingBox/20161015_20160425.json
./productBoundingBox/20161015_20160706.json <--
./productBoundingBox/20161015_20160706.json <--
./productBoundingBox/20161015_20160730.json <--
./productBoundingBox/20161015_20160730.json <--
./productBoundingBox/20161015_20160823.json <--
./productBoundingBox/20161015_20160823.json <--
./productBoundingBox/20161015_20160916.json <--
./productBoundingBox/20161015_20160916.json <--
./productBoundingBox/20161016_20160425.json
./productBoundingBox/20161016_20160706.json
./productBoundingBox/20161016_20160730.json
./productBoundingBox/20161108_20160425.json
./productBoundingBox/20161108_20160519.json <--
./productBoundingBox/20161108_20160519.json <--
./productBoundingBox/20161108_20160730.json <--
./productBoundingBox/20161108_20160730.json <--
./productBoundingBox/20161108_20160823.json <--
./productBoundingBox/20161108_20160823.json <--
./productBoundingBox/20161108_20160916.json <--
./productBoundingBox/20161108_20160916.json <--
./productBoundingBox/20161108_20161015.json <--
./productBoundingBox/20161108_20161015.json <--
./productBoundingBox/20161109_20160519.json
./productBoundingBox/20161109_20160730.json
./productBoundingBox/20161109_20160823.json
. . ., etc.

Is this expected behavior? Could this mean that not all scenes with gaps or duplicates are removed during ARIA_standardproduct.continuous_time step?

rzinke · Answer 3 · Thu Jun 23 2022 08:34:02 GMT+0800 (China Standard Time)

When I specify a much smaller minimum overlap area (-mo 25000, i.e, 1/10th the previous area), the script runs to completion.

ariaPlot.py -f "products/*.nc" -b ./AOI/track_048_TibetDii.shp -plottracks -nt 4 --figwidth wide -mo 25000 -w Test25K

Here are the outputs:

Scene: Test25/productBoundingBox/20160424_20160331.json
	area: 29065 - OK
Scene: Test25/productBoundingBox/20160425_20160401.json
	area: 670 - rejected
Scene: Test25/productBoundingBox/20160519_20160424.json
	area: 29060 - OK
Scene: Test25/productBoundingBox/20160612_20160331.json
	area: 29060 - OK
Scene: Test25/productBoundingBox/20160612_20160519.json
	area: 29060 - OK
Scene: Test25/productBoundingBox/20160706_20160401.json
	area: 648 - rejected
Scene: Test25/productBoundingBox/20160706_20160424.json
	area: 29060 - OK
Scene: Test25/productBoundingBox/20160706_20160612.json
	area: 29060 - OK
Scene: Test25/productBoundingBox/20160730_20160425.json
	area: 678 - rejected
Scene: Test25/productBoundingBox/20160730_20160519.json
	area: 29032 - OK
Scene: Test25/productBoundingBox/20160823_20160519.json
	area: 29032 - OK
Scene: Test25/productBoundingBox/20160823_20160612.json
	area: 29032 - OK
Scene: Test25/productBoundingBox/20160823_20160706.json
	area: 29032 - OK
Scene: Test25/productBoundingBox/20160823_20160730.json
	area: 29032 - OK
Scene: Test25/productBoundingBox/20160916_20160331.json
	area: 29032 - OK
Scene: Test25/productBoundingBox/20160916_20160519.json
	area: 29032 - OK
Scene: Test25/productBoundingBox/20160916_20160612.json
	area: 29032 - OK
Scene: Test25/productBoundingBox/20160916_20160706.json
	area: 29032 - OK
Scene: Test25/productBoundingBox/20160916_20160823.json
	area: 29032 - OK
Scene: Test25/productBoundingBox/20161015_20160401.json
	area: 647 - rejected
Scene: Test25/productBoundingBox/20161015_20160424.json
	area: 29031 - OK
Scene: Test25/productBoundingBox/20161015_20160425.json
	area: 646 - rejected
Scene: Test25/productBoundingBox/20161015_20160706.json
	area: 29031 - OK
Scene: Test25/productBoundingBox/20161015_20160706.json
	area: 29031 - OK
Scene: Test25/productBoundingBox/20161015_20160730.json
	area: 29031 - OK
Scene: Test25/productBoundingBox/20161015_20160730.json
	area: 29031 - OK
Scene: Test25/productBoundingBox/20161015_20160823.json
	area: 29031 - OK
Scene: Test25/productBoundingBox/20161015_20160823.json
	area: 29031 - OK
Scene: Test25/productBoundingBox/20161015_20160916.json
	area: 29031 - OK
Scene: Test25/productBoundingBox/20161015_20160916.json
	area: 29031 - OK
Scene: Test25/productBoundingBox/20161016_20160425.json
Scene: Test25/productBoundingBox/20161016_20160706.json
Scene: Test25/productBoundingBox/20161016_20160730.json
Scene: Test25/productBoundingBox/20161108_20160425.json
	area: 647 - rejected
Scene: Test25/productBoundingBox/20161108_20160519.json
	area: 29031 - OK
Scene: Test25/productBoundingBox/20161108_20160519.json
	area: 29031 - OK
Scene: Test25/productBoundingBox/20161108_20160730.json
	area: 29031 - OK
Scene: Test25/productBoundingBox/20161108_20160730.json
	area: 29031 - OK
Scene: Test25/productBoundingBox/20161108_20160823.json
	area: 29031 - OK
Scene: Test25/productBoundingBox/20161108_20160823.json
	area: 29031 - OK
Scene: Test25/productBoundingBox/20161108_20160916.json
	area: 29031 - OK
Scene: Test25/productBoundingBox/20161108_20160916.json
	area: 29031 - OK
Scene: Test25/productBoundingBox/20161108_20161015.json
	area: 29031 - OK
Scene: Test25/productBoundingBox/20161108_20161015.json
	area: 29031 - OK
Scene: Test25/productBoundingBox/20161109_20160519.json
Scene: Test25/productBoundingBox/20161109_20160730.json
Scene: Test25/productBoundingBox/20161109_20160823.json
Scene: Test25/productBoundingBox/20161109_20161016.json
Scene: Test25/productBoundingBox/20161202_20160331.json
	area: 29033 - OK
Scene: Test25/productBoundingBox/20161202_20160519.json
	area: 29033 - OK
Scene: Test25/productBoundingBox/20161202_20160519.json
	area: 29033 - OK
Scene: Test25/productBoundingBox/20161202_20160612.json
	area: 29033 - OK
Scene: Test25/productBoundingBox/20161202_20160612.json
	area: 29033 - OK
Scene: Test25/productBoundingBox/20161202_20160823.json
	area: 29033 - OK
Scene: Test25/productBoundingBox/20161202_20160823.json
	area: 29033 - OK
Scene: Test25/productBoundingBox/20161202_20160915.json
	area: 29033 - OK
Scene: Test25/productBoundingBox/20161202_20160916.json
	area: 29033 - OK
Scene: Test25/productBoundingBox/20161202_20160916.json
	area: 29033 - OK
Scene: Test25/productBoundingBox/20161202_20161108.json
	area: 29033 - OK
Scene: Test25/productBoundingBox/20161202_20161108.json
	area: 29033 - OK
Scene: Test25/productBoundingBox/20161203_20160331.json
	area: 665 - rejected
Scene: Test25/productBoundingBox/20161203_20160519.json
Scene: Test25/productBoundingBox/20161203_20160823.json
Scene: Test25/productBoundingBox/20161203_20160916.json
	area: 665 - rejected
Scene: Test25/productBoundingBox/20161203_20161109.json
Scene: Test25/productBoundingBox/20161226_20160612.json
	area: 29033 - OK
Scene: Test25/productBoundingBox/20161226_20160612.json
	area: 29033 - OK
Scene: Test25/productBoundingBox/20161226_20160706.json
	area: 29033 - OK
Scene: Test25/productBoundingBox/20161226_20160706.json
	area: 29033 - OK
Scene: Test25/productBoundingBox/20161226_20160916.json
	area: 29033 - OK
Scene: Test25/productBoundingBox/20161226_20160916.json
	area: 29033 - OK
Scene: Test25/productBoundingBox/20161226_20161015.json
	area: 29033 - OK
Scene: Test25/productBoundingBox/20161226_20161015.json
	area: 29033 - OK
Scene: Test25/productBoundingBox/20161226_20161202.json
	area: 29033 - OK
Scene: Test25/productBoundingBox/20161226_20161202.json
	area: 29033 - OK
Scene: Test25/productBoundingBox/20161227_20160612.json
Scene: Test25/productBoundingBox/20161227_20160706.json
Scene: Test25/productBoundingBox/20161227_20160916.json
Scene: Test25/productBoundingBox/20161227_20161016.json
Scene: Test25/productBoundingBox/20161227_20161203.json
23 out of 80 interferograms rejected for not meeting specified spatial thresholds
- Make plot of track latitude extents vs bounding bbox/common track extent.

Notice how none of the date pairs that appear twice (e.g., 20161015_20160706) are rejected this time. So maybe merged_productbbox is having trouble recording the rejected scenes.

BB · Answer 4 · Thu Jul 28 2022 09:18:03 GMT+0800 (China Standard Time)

Fixed with #313, thanks @sssangha!