[SEN-ET_automatization]"NameError: name 'valid_list_s2_images' is not defined" during code run

Question

[SEN-ET_automatization]"NameError: name 'valid_list_s2_images' is not defined" during code run

coomomo opened this issue 2 years ago · comments

coomomo · Answer 1 · Wed Nov 23 2022 20:59:58 GMT+0800 (China Standard Time)

Hello, I encountered "NameError: name 'valid_list_s2_images' is not defined" when running the code "0_path_read.py". I don't know how to fix this error, I hope I can get some help, thanks!

Lorenzo Stucchi · Answer 2 · Thu Nov 24 2022 21:34:17 GMT+0800 (China Standard Time)

Dear @coomomo, the error seems connected to the fact that there are no Sentinel-2 images in the indicated directory. Did you follow the steps indicated in the README.md and download the satellite images?

coomomo · Answer 3 · Thu Nov 24 2022 22:43:13 GMT+0800 (China Standard Time)

Thank you very much for your answer. I have solved this problem now, the reason for this problem was that I had unrecognized characters in the folder name. But now I seem to have another problem😣, in "2_read_times_s3.py", "for day in date_s3: path = 'E:\\\TEMP\\\MAIN_FOLDER\\' + "s3\\\" + day + "\\\S3_" + day + "_mask.dim" for line in open(path, 'r').readlines(): if "PRODUCT_SCENE_RASTER_START_TIME" in line: data_start = line.split(">")[1].split("<")[0]" The time obtained by this code is the time of S2 and not S3, I have tried to modify it many times, but still no success. By the way, I would like to ask you if you are writing in IOS system? I hope to get your reply. Thanks!😀 Lorenzo Stucchi ***@***.***> 于2022年11月24日周四 21:34写道：

…

Dear @coomomo <https://github.com/coomomo>, the error seems connected to the fact that there are no Sentinel-2 images in the indicated directory. Did you follow the steps indicated in the README.md <https://github.com/LorenzoStucchi/SEN-ET_automatization#folder-structure> and download the satellite images? — Reply to this email directly, view it on GitHub <#6 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWNCJ7FEJTSN246NWQNPCQ3WJ5VGLANCNFSM6AAAAAASI52THI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

Lorenzo Stucchi · Answer 4 · Fri Nov 25 2022 00:04:34 GMT+0800 (China Standard Time)

As before I believe this should be an issue with the path, try to check if the path exists.

It is searching the file connected with the S3 images and only that, so maybe your file are in the wrong location

coomomo · Answer 5 · Fri Nov 25 2022 23:12:07 GMT+0800 (China Standard Time)

I solved the S3 time problem yesterday, thanks for your guidance.😀 But now 0_path_read.py is wrong again, in lines 15-18 of "0_path_read.py" sub_folders = [name for name in os.listdir(general_path) if os.path.isdir(os.path.join(general_path, name))] for folder in sub_folders: temp_path = general_path + "\\" + folder Since " general_path=E:\\TEMP\\ and sub_folders=MAIN_FOLDER, why is folder still equal to MAIN_FOLDER? I print(temp_path), and the result is "E:\TEMP\\\MAIN_FOLDER", not "E:\TEMP\\\MAIN_FOLDER\\\s2" or "E:\TEMP\\\MAIN_FOLDER\s3". I think the temp_path should be equal to "E:\TEMP\\\MAIN_FOLDER\\\s2" in order to read the s2 images, now the code error is "valid_list_s2_images ""valid_list_s3 images" does not exist. But I checked the code in lines 1-14, and I think it is correct. I hope I can get your help again, I've been struggling with this problem for a long time and can't solve it.😣 I've put a screenshot of my code in the attachment, thank you very much again! Lorenzo Stucchi ***@***.***> 于2022年11月25日周五 00:04写道：

…

As before I believe this should be an issue with the path, try to check if the path exists. It is searching the file connected with the S3 images and only that, so maybe your file are in the wrong location — Reply to this email directly, view it on GitHub <#6 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWNCJ7FHSA3MGU7ETYPRMNLWJ6GZ3ANCNFSM6AAAAAASI52THI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

Lorenzo Stucchi · Answer 6 · Thu Dec 01 2022 23:22:41 GMT+0800 (China Standard Time)

I cannot see your image, try to attach them on github issue and not in the email

coomomo · Answer 7 · Tue Dec 06 2022 21:25:32 GMT+0800 (China Standard Time)

Thank you very much for your previous answer, I have solved the path problem. But there are still several problems in the program that I have tried many times and still can't solve.I hope I can get your help，Thanks again.

` Processing the image S2 2021_01_18
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL not found on system. Internal GDAL 3.0.0 from distribution will be used. (f1)
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
Executing processing graph
INFO: org.esa.s2tbx.dataio.s2.ortho.S2OrthoProductReaderPlugIn: Building product reader - EPSG:32648
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [metadata_level] does not exist
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist
INFO: org.hsqldb.persist.Logger: dataFileCache open start
WARNING: org.esa.s2tbx.dataio.s2.ortho.metadata.S2OrthoMetadata: Warning: missing file E:\TEMP\MAIN_FOLDER\s2\2021_01_18\S2B_MSIL2A_20210118T033059_N0214_R018_T48QXM_20210118T062400.SAFE\GRANULE\L2A_T48QXM_A020207_20210118T033053\QI_DATA\L2A_T48QXM_20210118T033059_DDV_20
m.jp2

WARNING: org.esa.s2tbx.dataio.s2.ortho.metadata.S2OrthoMetadata: Warning: no image files found for band quality_dense_dark_vegetation

SEVERE: org.esa.s2tbx.biophysical.BiophysicalAuxdata: Error when reading C:\Users\Administrator.snap\auxdata\s2tbx\biophysical\3_0\S2B\LAI_Cw\LAI_Cw_TestCases
WARNING: org.esa.s2tbx.biophysical.BiophysicalAuxdata: Error when loading coefficients TEST_CASES for variable LAI_Cw model S2B. They won't be available.`

Sharpening the image S3 2021_01_18 03:03 INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL 3.0.2 found on system. JNI driver will be used. INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 3.0.2 set to be used by SNAP. INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience. INFO: Deriving solar illumination conditions... INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 3.0.2 set to be used by SNAP. INFO: org.hsqldb.persist.Logger: dataFileCache open start INFO: org.esa.snap.dataio.bigtiff.BigGeoTiffProductWriter: writing to output file C:\Users\ADMINI~1\AppData\Local\Temp\tmph1zuweee.tif Saved C:\Users\ADMINI~1\AppData\Local\Temp\tmph1zuweee_cos_theta.tif INFO: Preparing high-resolution data... INFO: org.esa.snap.dataio.bigtiff.BigGeoTiffProductWriter: writing to output file C:\Users\ADMINI~1\AppData\Local\Temp\tmp_lsb7_ij.tif Warning 1: TIFFReadDirectory:Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples. Warning 1: TIFFReadDirectory:Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples. Warning 1: TIFFReadDirectory:Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples. Warning 1: TIFFReadDirectory:Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples. Warning 1: TIFFReadDirectory:Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples. Warning 1: TIFFReadDirectory:Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples. Warning 1: TIFFReadDirectory:Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples. Warning 1: TIFFReadDirectory:Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples. Warning 1: TIFFReadDirectory:Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples. Warning 1: TIFFReadDirectory:Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples. INFO: org.esa.snap.dataio.bigtiff.BigGeoTiffProductWriter: writing to output file C:\Users\ADMINI~1\AppData\Local\Temp\tmpinzg6xr1.tif INFO: org.esa.snap.dataio.bigtiff.BigGeoTiffProductWriter: writing to output file C:\Users\ADMINI~1\AppData\Local\Temp\tmpn1tefd__.tif INFO: Training regressor... Traceback (most recent call last): File "C:\Users\Administrator\.snap\auxdata\sen-et-conda-Win64\sen-et-snap-scripts\data_mining_sharpener.py", line 216, in <module> main() File "C:\Users\Administrator\.snap\auxdata\sen-et-conda-Win64\lib\site-packages\click\core.py", line 764, in __call__ return self.main(*args, **kwargs) File "C:\Users\Administrator\.snap\auxdata\sen-et-conda-Win64\lib\site-packages\click\core.py", line 717, in main rv = self.invoke(ctx) File "C:\Users\Administrator\.snap\auxdata\sen-et-conda-Win64\lib\site-packages\click\core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "C:\Users\Administrator\.snap\auxdata\sen-et-conda-Win64\lib\site-packages\click\core.py", line 555, in invoke return callback(*args, **kwargs) File "C:\Users\Administrator\.snap\auxdata\sen-et-conda-Win64\sen-et-snap-scripts\data_mining_sharpener.py", line 97, in main disaggregator.trainSharpener() File "C:\Users\Administrator\.snap\auxdata\sen-et-conda-Win64\lib\site-packages\pyDMS\pyDMS.py", line 332, in trainSharpener qualityPix = np.logical_and(qualityPix, ~np.isnan(data_LR)) ValueError: operands could not be broadcast together with shapes (5490,5490) (96,96)

coomomo · Answer 8 · Tue Dec 06 2022 22:17:13 GMT+0800 (China Standard Time)

Although "WARNING: org.esa.s2tbx.biophysical.BiophysicalAuxdata: Error when loading coefficients TEST_CASES for variable LAI_Cw model S2B. They won't be available." appears, but this process has result output, I'm not sure if the result is correct. And "Sharpen LST" has no result output

coomomo · Answer 9 · Tue Dec 06 2022 22:45:22 GMT+0800 (China Standard Time)

I am using windows 10, snap8.0, python3.4

Lorenzo Stucchi · Answer 10 · Tue Dec 06 2022 23:19:26 GMT+0800 (China Standard Time)

Although "WARNING: org.esa.s2tbx.biophysical.BiophysicalAuxdata: Error when loading coefficients TEST_CASES for variable LAI_Cw model S2B. They won't be available." appears, but this process has result output, I'm not sure if the result is correct. And "Sharpen LST" has no result output

This is a SNAP warning, it is ok and the results is valid

coomomo · Answer 11 · Wed Dec 07 2022 11:43:26 GMT+0800 (China Standard Time)

Although "WARNING: org.esa.s2tbx.biophysical.BiophysicalAuxdata: Error when loading coefficients TEST_CASES for variable LAI_Cw model S2B. They won't be available." appears, but this process has result output, I'm not sure if the result is correct. And "Sharpen LST" has no result output

This is a SNAP warning, it is ok and the results is valid

Thank you. "ValueError: operands could not be broadcast together with shapes" when I try to run "Sharpen LST". How should I correct this mistake?

Lorenzo Stucchi · Answer 12 · Wed Dec 07 2022 14:42:26 GMT+0800 (China Standard Time)

Thank you. "ValueError: operands could not be broadcast together with shapes" when I try to run "Sharpen LST". How should I correct this mistake?

Sorry, I don't have enough information to provide help. Do you insert all the correct inputs?

Lorenzo Stucchi · Answer 13 · Tue Jan 03 2023 19:08:06 GMT+0800 (China Standard Time)

@coomomo did you solve the found issue? Do you need more help?

coomomo · Answer 14 · Mon Jan 09 2023 23:17:29 GMT+0800 (China Standard Time)

@LorenzoStucchi
The problem I encountered regarding "ValueError: operands could not be broadcast together with shapes (5490,5490) (96,96)" has been solved. My solution was to manually operate in SNAP when doing "warp to template", "sharpen LST" and the input file must be "graph" The input file must be the "LST", "MASK" and "observation geometry" from the "graph" folder of sentinel_3_pre_processing.xml. I think the reason for this one error may be that the study area is not in the right range.
Thank you again for your previous help.