Problems processing S3 Images

Question

Problems processing S3 Images

FranGutierrezC opened this issue 9 months ago · comments

Hi everyone! I have some troubles running the processing for S3 images and I would appreciate some help. Before the execution of the .sh, I updated the sen-et-scripts (https://github.com/DHI-GRAS/sen-et-snap-scripts) and cheked that the graphs of the graphs folder were the corrected ones.
When I execute de sh main.sh file from git bash I got all the outputs from S2, however for the S3 data, I only receive the observation geometry, lst and mask. In the Output folder “out” generated, the only output file obtained is the longwave_ir. Also, the folder to save the ET.tiff also is missing. As I said before, im troubling trying to solve the issue. I copy the part of the error for one of the images that im using.

Im using python 3.6, and working with "S2B_MSIL2A_20221204T143729_N0400_R096_T19KDP_20221204T183146" and "S3B_SL_2_LST____20221202T141651_20221202T141951_20221203T124713_0179_073_224_3420_PS2_O_NT_004"

I will be attentive to any questions or additional information that is necessary to resolve this problem, I greatly appreciate your time.

(…)
Warp the image S3 2022_12_02 14:17
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
Usage: warp_to_template.py [OPTIONS]
Try "warp_to_template.py --help" for help.

Error: Invalid value for "--template": File "D:\PROYECTO_B\S2\2022_12_04\S2_2022_12_04_resampled.dim" does not exist.
Sharpening the image S3 2022_12_02 14:17
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
Usage: data_mining_sharpener.py [OPTIONS]
Try "data_mining_sharpener.py --help" for help.

Error: Invalid value for "--high_res_geom": File "D:\PROYECTO_B\S3\2022_12_02\S3_2022_12_02_hr_vza.dim" does not exist.
Computing the meteo data for the day 2022_12_02 14:17
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.hsqldb.persist.Logger: dataFileCache open start
INFO: org.esa.snap.dataio.bigtiff.BigGeoTiffProductWriter: writing to output file C:\Users\AETOS~1.I7-\AppData\Local\Temp\tmpyl8kes3z.tif
Warning 1: Recode from UTF-8 to CP_ACP failed with the error: "Invalid argument".
Warning 1: Recode from UTF-8 to CP_ACP failed with the error: "Invalid argument".
Warning 1: Recode from UTF-8 to CP_ACP failed with the error: "Invalid argument".
Warning 1: Recode from UTF-8 to CP_ACP failed with the error: "Invalid argument".
Warning 1: Recode from UTF-8 to CP_ACP failed with the error: "Invalid argument".
Warning 1: Recode from UTF-8 to CP_ACP failed with the error: "Invalid argument".
Warning 1: Recode from UTF-8 to CP_ACP failed with the error: "Invalid argument".
Warning 1: Recode from UTF-8 to CP_ACP failed with the error: "Invalid argument".
Warning 1: Recode from UTF-8 to CP_ACP failed with the error: "Invalid argument".
Warning 1: Recode from UTF-8 to CP_ACP failed with the error: "Invalid argument".
air_temperature
Saved MEM
Saved MEM
Saved MEM
vp
Saved MEM
ap
Saved MEM
ws
Saved MEM
cssr
Saved MEM
adsi
Saved MEM
Computing the longwave radiation for the image S3 2022_12_02 14:17
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.hsqldb.persist.Logger: dataFileCache open start
Computing the shortwave radiation for the image S3 2022_12_02 14:17
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
Usage: net_shortwave_radiation.py [OPTIONS]
Try "net_shortwave_radiation.py --help" for help.

Error: Invalid value for "--sza_product": File "D:\PROYECTO_B\S3\2022_12_02\S3_2022_12_02_hr_vza.dim" does not exist.
Computing the energy fluxes for the image S3 2022_12_02 14:17
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
Usage: energy_fluxes.py [OPTIONS]
Try "energy_fluxes.py --help" for help.

Error: Invalid value for "--lst": File "D:\PROYECTO_B\S3\2022_12_02\S3_2022_12_02_lst_sharpened.dim" does not exist.
Computing the evapotranspiration for the image S3 2022_12_02 14:17
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
Usage: daily_evapotranspiration.py [OPTIONS]
Try "daily_evapotranspiration.py --help" for help.

Error: Invalid value for "--ief_file": File "D:\PROYECTO_B\out\2022_12_02\2022_12_02_instantaneous_fluxes.dim" does not exist.
Transformation of the evapotranspiration for the image S3 2022_12_02 14:17
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
Executing processing graph
done.

Error: [NodeId: Read] Specified 'file' [D:\PROYECTO_B\et\dim\2022_12_02_daily_evapotranspiration.dim] does not exist.
Finish the computation of the evapotranspiration for the image S3 2022_12_02 14:17

automatization_sen-et_issue.txt

Lorenzo Stucchi · Answer 1 · Mon Oct 02 2023 14:47:25 GMT+0800 (China Standard Time)

Hi, the first error in the log is the code of 1_script_gpt_s3.sh, here:

         Processing the image S3 2022_12_02
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
Executing processing graph
WARNING: org.esa.s3tbx.dataio.s3.AbstractProductFactory: D:\PROYECTO_B\S3\2022_12_02\S3B_SL_2_LST____20221202T141651_20221202T141951_20221203T124713_0179_073_224_3420_PS2_O_NT_004.SEN3 (Acceso denegado)
WARNING: org.esa.s3tbx.dataio.s3.AbstractProductFactory: Could not find ''.
 done.

Error: [NodeId: Subset] Operator 'SubsetOp': Value for 'Source Bands' is invalid: 'sat_zenith_tn'

I'm not understanding this part of your previous text:

Before the execution of the .sh, I updated the sen-et-scripts (https://github.com/DHI-GRAS/sen-et-snap-scripts) and cheked that the graphs of the graphs folder were the corrected ones.

It is not needed to change nothing, the correct graph are the one present in the folder graph of this repo

FranGutierrezC · Answer 2 · Mon Oct 02 2023 18:23:41 GMT+0800 (China Standard Time)

Thanks for your reply!
I notice this error off "access denied" to all the folders of sentinel-3 imagen. However I've cheked that the folders had the permision, tried to use the chmod +x to give complete access and also I've been execute gitbash as administrator, but none of this have work. Have you any ideas to solve the issue? What I cant understand is that if the folder of S3 had access denied, why I can obtein the products of lst, mask and observed geometry?
About the scripts, I updated the ones of the sen-et-scripts folders because ive found some difference with the ones in this repo (https://github.com/DHI-GRAS/sen-et-snap-scripts)

Lorenzo Stucchi · Answer 3 · Mon Oct 02 2023 18:38:15 GMT+0800 (China Standard Time)

Yes, there are differences between the scripts, but the onesin this repo are the ones for this code. Try to do it with the scripts in this repo
I made the change in the other repo because they were wrong https://github.com/DHI-GRAS/sen-et-snap-scripts/commits?author=LorenzoStucchi

FranGutierrezC · Answer 4 · Mon Oct 02 2023 19:02:15 GMT+0800 (China Standard Time)

Ok thanks, I'll make a test with those scripts.
About the access denied to the folder, do you have any ideas to solve it? or had you this similar problem before? i've trying to solve it but nothing works

Lorenzo Stucchi · Answer 5 · Mon Oct 02 2023 19:53:03 GMT+0800 (China Standard Time)

It could be an issue that is not critical, let's try with the scripts if they solve the issue.

FranGutierrezC · Answer 6 · Mon Oct 02 2023 22:10:44 GMT+0800 (China Standard Time)

Hi! I've try the solution by replacing the scripts of the link that you gave me, however im still getting the same error of access denied to the folders .SEN3 of the images that I'm using. I also attach the output "4 script S3" to check if there's a problem in the generation of the outputs.

Aetos.i7-10700@Fran-geolab MINGW64 /d/SEN-ET_automatization (master)
$ sh main.sh
Path files created!
input/s2_paths.txt
input/s3_paths.txt

real 0m0.138s
user 0m0.000s
sys 0m0.000s

    Scripts created!
    output\1_script_gpt_s3.sh
    output\2_script_gpt_s2.sh

real 0m0.171s
user 0m0.000s
sys 0m0.000s

     Processing the image S3 2022_12_02

INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
Executing processing graph
WARNING: org.esa.s3tbx.dataio.s3.AbstractProductFactory: D:\Proyecto_B\S3\2022_12_02\S3B_SL_2_LST____20221202T141651_20221202T141951_20221203T124713_0179_073_224_3420_PS2_O_NT_004.SEN3 (Access denied)
WARNING: org.esa.s3tbx.dataio.s3.AbstractProductFactory: Could not find ''.
INFO: org.hsqldb.persist.Logger: dataFileCache open start
.13%.27%..38%.48%..62%..76%.90% done.

4_script_S3.txt

Lorenzo Stucchi · Answer 7 · Tue Oct 03 2023 17:05:06 GMT+0800 (China Standard Time)

It looks that the access is denied but then the comand is done. The product is created?
The 4_script_S3.txt is the same as one that I have as example.
Which version of snap are you using?

Lorenzo Stucchi · Answer 8 · Tue Oct 03 2023 22:53:59 GMT+0800 (China Standard Time)

I tested the code only on snap 8, maybe could be something related to snap 9, I did not tested on this version

FranGutierrezC · Answer 9 · Tue Oct 03 2023 23:32:25 GMT+0800 (China Standard Time)

Trying to find the issue, i've notice that the script .xml of sentinel_2_preprocessing doesn't save the output of the resample of the bands, is that correct or is not necesary in the workflow? Because i check that in the "warp_to_template" starts mi problem. And I got all de inputs necesary except for the resample.dim file . When I checked and open the .xml file from snap I realized that theres no box in the graph builder that writes the "Write_resample" as with the other products of S2. I think that having that file _resample.dim I could get the others missing outputs. I would appreciate some guidance to solve the issue. Thanks so much for your time so far

sentinel_2_pre_processing_xml.txt

Lorenzo Stucchi · Answer 10 · Wed Oct 04 2023 14:25:26 GMT+0800 (China Standard Time)

Ok, found the issue, the variable have been rename but not all the changes have been uploaded, I have added them now.

So now, the warp_to_template.py takes 3 input and provides 1 output:

--source: that is obs_geometry.dim that comes from the sentinel_3_pre_processing.xml at the node Write_obs_geometry
--template: that is mask.dim that comes from the sentinel_2_pre_processing.xml at the node Write_mask
--resample_algoritm: that is directly set to near

Now, it should work

Lorenzo Stucchi · Answer 11 · Wed Oct 04 2023 14:48:18 GMT+0800 (China Standard Time)

fix in 165e2c1

Lorenzo Stucchi · Answer 12 · Fri Oct 06 2023 17:21:45 GMT+0800 (China Standard Time)

Thank you for the help in understanding the problem!

As you said, there is a lack of info on the documentation. You could try to contact the authors directly via mail.

However, as far as I remember, I believe that the lake has been removed because it has been recognized as water, and the model does not compute direct ET from water.

Lorenzo Stucchi · Answer 13 · Thu Oct 19 2023 22:00:07 GMT+0800 (China Standard Time)

I don't know if have you ever experience a similar problem with multitemporal analysis of images, any guidane will help me. Thanks again, and my apologize for the insistence.

My usage was for computing a year of data. So I wrote the code to choose the S2 image closest to the S3 of reference. You could try to provide fewer S2 images and see how these affect the results.

Also, the usage of S2A and S2B could provide different results, maybe staying just on one of the two. Same for S3A and S3B

FranGutierrezC · Answer 14 · Fri Oct 20 2023 21:37:12 GMT+0800 (China Standard Time)

Thanks for your answer Lorenzo, now it makes sense. I'll work with S2A because those images are the ones that gave me the output that i expected, and I will select only one sensor for S3. I appreciate your help

Problems processing S3 Images

real 0m0.138s user 0m0.000s sys 0m0.000s

real 0m0.171s user 0m0.000s sys 0m0.000s

real 0m0.138s
user 0m0.000s
sys 0m0.000s

real 0m0.171s
user 0m0.000s
sys 0m0.000s