L3 resampling

Question

L3 resampling

KapteinProton opened this issue 4 years ago · comments

Martin Album Ytre-Eide commented 4 years ago

I can not seem to get the converting into L3 to work. Downloading works greart, but after several tries I still get:

Converting into L3 products

Launched 8 processes
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/mye/anaconda3/envs/s5p/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/home/mye/evdc/s5p-tools/s5p_tools/preprocess.py", line 32, in _process_file
operations=pre_commands)
File "/home/mye/anaconda3/envs/s5p/lib/python3.7/site-packages/harp/_harppy.py", line 1187, in import_product
raise CLibraryError()
harp._harppy.CLibraryError: syntax error, unexpected ';', expecting $end
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "s5p-tools/s5p-request.py", line 306, in
num_workers=args.num_workers)
File "s5p-tools/s5p-request.py", line 151, in main
num_workers=num_workers)
File "/home/mye/evdc/s5p-tools/s5p_tools/preprocess.py", line 73, in convert_to_l3_products
desc="Converting", leave=False, total=len(filenames)))
File "/home/mye/anaconda3/envs/s5p/lib/python3.7/site-packages/tqdm/std.py", line 1174, in iter
for obj in iterable:
File "/home/mye/anaconda3/envs/s5p/lib/python3.7/multiprocessing/pool.py", line 748, in next
raise value
harp._harppy.CLibraryError: syntax error, unexpected ';', expecting $end

Bilel Omrani · Answer 1 · Tue Oct 20 2020 21:19:56 GMT+0800 (China Standard Time)

Should be solved, there was an issue with the way the harp command was generated for CH4 (and a few other) ingestion. Feel free to reopen the issue if it did not solve your issue.

Martin Album Ytre-Eide · Answer 2 · Tue Oct 20 2020 22:39:25 GMT+0800 (China Standard Time)

I get an new error now:
Converting into L3 products

Launched 8 processes
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/mye/anaconda3/envs/s5p/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/home/mye/evdc/s5p-tools/s5p_tools/preprocess.py", line 31, in _process_file
str(filename), operations=pre_commands
File "/home/mye/anaconda3/envs/s5p/lib/python3.7/site-packages/harp/_harppy.py", line 1187, in import_product
raise CLibraryError()
harp._harppy.CLibraryError: out of memory (could not allocate 62208000000 bytes) (/home/jenkins/workspace/CondaPackages/conda3-linux-64/miniconda3/conda-bld/harp_1589906707658/work/libharp/harp-binning.c:2305)
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "s5p-request.py", line 411, in
num_workers=args.num_workers,
File "s5p-request.py", line 217, in main
num_workers=num_workers,
File "/home/mye/evdc/s5p-tools/s5p_tools/preprocess.py", line 84, in convert_to_l3_products
total=len(filenames),
File "/home/mye/anaconda3/envs/s5p/lib/python3.7/site-packages/tqdm/std.py", line 1174, in iter
for obj in iterable:
File "/home/mye/anaconda3/envs/s5p/lib/python3.7/multiprocessing/pool.py", line 748, in next
raise value
harp._harppy.CLibraryError: out of memory (could not allocate 62208000000 bytes) (/home/jenkins/workspace/CondaPackages/conda3-linux-64/miniconda3/conda-bld/harp_1589906707658/work/libharp/harp-binning.c:2305)

Bilel Omrani · Answer 3 · Wed Oct 21 2020 00:56:23 GMT+0800 (China Standard Time)

Can you describe what are you querying exactly? What is the size of your area of interest, and what is the target resolution of your grid? It seems that you are trying to allocate to much memory during the resampling.

Martin Album Ytre-Eide · Answer 4 · Wed Oct 21 2020 06:33:44 GMT+0800 (China Standard Time)

I have tried both with and without aoi, and without it seems to be a memory problem yes. With an aoi I get:

python s5p-request.py L2__CH4___ --aoi map.geojson --date 20200723 20200724

Requesting products

Number of products found: 11
Total products size: 0.59 GB

Launched 4 threads
File a13a36a5-10e7-4b66-b0cc-35222994c426 already exists
File 307607cf-2b3d-44fc-bdcb-3faec9f003d8 already exists
File ea6bf061-f326-4093-b5d7-33bd86c78ccf already exists
File 0312c143-79a2-4337-ab7b-074ce592b4e4 already exists
File ca930abe-adb5-4593-be65-f1884496c0fd already exists
File c83c4143-8ecc-49f9-95c8-543cb0af73e0 already exists
File a50776bc-e677-48a3-b0c0-5cfee20af768 already exists
File 4acf027e-b2bb-4e36-8aba-28d4e2f40889 already exists
File 79ee7e2c-1f54-4583-8007-bd6e2e4585f3 already exists
File 522a7340-0862-42db-b097-a26884bf2bd7 already exists
File bec702b0-0b26-4638-8fd3-ffdbf63f59c6 already exists

Converting into L3 products

Launched 8 processes
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/mye/anaconda3/envs/s5p/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/home/mye/evdc/s5p-tools/s5p_tools/preprocess.py", line 31, in process_file
str(filename), operations=pre_commands
File "/home/mye/anaconda3/envs/s5p/lib/python3.7/site-packages/harp/harppy.py", line 1187, in import_product
raise CLibraryError()
harp.harppy.CLibraryError: L2_data/L2__CH4/S5P_OFFL_L2__CH4____20200723T141133_20200723T155303_14389_01_010302_20200730T072921.nc: unsupported product
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "s5p-request.py", line 411, in
num_workers=args.num_workers,
File "s5p-request.py", line 217, in main
num_workers=num_workers,
File "/home/mye/evdc/s5p-tools/s5p_tools/preprocess.py", line 84, in convert_to_l3_products
total=len(filenames),
File "/home/mye/anaconda3/envs/s5p/lib/python3.7/site-packages/tqdm/std.py", line 1174, in iter
for obj in iterable:
File "/home/mye/anaconda3/envs/s5p/lib/python3.7/multiprocessing/pool.py", line 748, in next
raise value
harp.harppy.CLibraryError: L2_data/L2__CH4__/S5P_OFFL_L2__CH4____20200723T141133_20200723T155303_14389_01_010302_20200730T072921.nc: unsupported product

If you have the geojson/france.geojson file, I could try to reproduce the example.

Bilel Omrani · Answer 5 · Wed Oct 21 2020 07:45:32 GMT+0800 (China Standard Time)

You can use http://geojson.io/ to generate custom AOIs but I doubt that this is the issue. The traceback seems weird, and I cannot reproduce it on my machine. Few tips:

Try to empty the cache by deleting the L2 and L3 directories
You seem to query a global coverage of CH4 at default resolution (0.01x0.01 arc degrees resolution). That's usually not going to work (except if you have an insane amount of RAM) because the whole file has to be first generated in memory. Try to adjust the resolution of the grid with the --resolution flag. Alternatively, if you still need to work at finest resolution, chunk your AOI in multiple pieces (spatially and/or temporally) and use Dask or other lazy-loaded arrays libraries to work on these multiple files during your analysis.

Please file a new issue in case something is not working with the script.

Martin Album Ytre-Eide · Answer 6 · Wed Oct 21 2020 17:07:06 GMT+0800 (China Standard Time)

Combination of AOI and resolution made the script work. I will have a look at the result now. I had to reduce the resolution quite much, but as you said, I could try to tile my AOI. For my purpose I could also do with a low resolution, and after inspection do higher resolution on hot-spots with smaller AOI. Thanks!

Hichem Omrani · Answer 7 · Wed Oct 21 2020 21:58:28 GMT+0800 (China Standard Time)

?awesome :) Cheers

…

--- Follow LISER on Facebook<https://www.facebook.com/LISERinLUX>, Twitter<https://twitter.com/LISERinLUX> and LinkedIn<https://www.linkedin.com/company/liser---luxembourg-institute-of-socio-economic-research>

________________________________ De : Martin Album Ytre-Eide <notifications@github.com> Envoyé : mercredi, le 21 octobre 2020 11:07 À : bilelomrani1/s5p-analysis Cc : hichem omrani; Comment Objet : Re: [bilelomrani1/s5p-analysis] L3 resampling (#2) EXTERNAL SENDER - Be CAUTIOUS with links and attachments - DO NOT ENTER YOUR CREDENTIALS.

________________________________ Combination of AOI and resolution made the script work. I will have a look at the result now. I had to reduce the resolution quite much, but as you said, I could try to tile my AOI. For my purpose I could also do with a low resolution, and after inspection do higher resolution on hot-spots with smaller AOI. Thanks! - You are receiving this because you commented. Reply to this email directly, view it on GitHub<#2 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADXJTQGSPHUQL2UO6OH5EULSL2QEXANCNFSM4SX35BJA>. This message (including any attachments) is intended for the addressee only and may contain confidential and/or privileged information and/or information protected by intellectual property rights. If you have received this message by mistake, please notify the sender by return e-mail and delete this message from your system. You should not use, alter, copy or distribute this message or disclose its contents to anyone. E-mail transmission can not be guaranteed to be secure or error free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. Luxembourg Institute of Socio-economic Research shall not be responsible nor liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt or damage to your system.