Giters

bilelomrani1 / s5p-tools

Python scripts to download and preprocess air pollution concentration level data aquired from the Sentinel-5P mission

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

L3 Resampling and memory usage

3zero2 opened this issue · comments

commented

Is it normal that the HARP tools for the L3 resampling use huge amounts of memory? It tried to download and process L2__NO2___ data with this command:

python s5p-request.py L2__NO2___ --date NOW-2HOUR NOW --num-workers 1

The download was quite small but when it got to the L3 resampling python started to use huge amounts of RAM until it crashed the computer. Is this expected when using HARP tools?

Screenshot 2021-03-08 at 18 57 36

commented

You did not specify an area of interest in the above command, so by default the script will download and process the NO2 data over the whole globe. The default resolution being 7x4km approximately, this is indeed a huge amount of data to resample. As far as I know, HARP tries to load the whole grid in memory and cannot work in batch.
If you want to resample the data globally, i suggest either:

  • lowering the resolution with the --resolution option
  • splitting the area of interest in multiple smaller chunks
commented

You are completely right. I provided an --aoi and now it completes.

For those reading this I had to put import geopandas on the first line of the s5p_requests file. For some reason if geopandas was not imported first it was. giving off a segmentation fault.