Medoid compositing method always running out of memory?

[Vytautas Jancauskas]

I'm interested in this approach but whenever I try it it always runs out of memory (even though other compositing methods work fine with the same settings)

[Dugal Harris]

If the image computation is running over the user memory limit on download, you can reduce the max_tile_size parameter. I suggest trying max_tile_size=8 (MB). The command line docs are here and the API docs here.

[Vytautas Jancauskas]

Hi Dugal, 8 is what I have it set to after experimenting for a value that stops throwing those memory errors 99% of the time, I think something more is going on here. Does this method work for you?

[Dugal Harris]

Hello, I have seen a "user memory limit exceeded" error when downloading a large Sentinel-2 medoid composite. When I decreased max_tile_size, it downloaded ok.

Earth engine has a size limit (32 MB) on download requests. geedim calculates the download tile sizes so that this is not exceeded. Earth engine also has a limit on user memory for image computations. This limit can still be exceeded when downloading computed images (e.g. medoid composites) with geedim. Unfortunately there is no way for geedim to know the memory requirements of a computation in advance. So it is up to the user to adjust max_tile_size if they run into memory issues.

Alternatively, exporting is not subject to the user memory limit, so a user could export their computed image to an asset, and then download the asset image with geedim.

If you think there is something else going wrong, please provide code to recreate the problem case.

[Vytautas Jancauskas]

I will try with even smaller tile sizes when I get a chance (next week)

[Dugal Harris]

See the related documentation section in the latest release.

[Vytautas Jancauskas]

Well, for me it doesn't work even if I reduce max_tile_size to 0.1

[Vytautas Jancauskas]

Here is the GeoJSON I've used.

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {},
      "geometry": {
        "coordinates": [
          [
            [
              -0.3689373648039975,
              5.785841065493003
            ],
            [
              -0.3689373648039975,
              5.364440103464588
            ],
            [
              0.05576223573189054,
              5.364440103464588
            ],
            [
              0.05576223573189054,
              5.785841065493003
            ],
            [
              -0.3689373648039975,
              5.785841065493003
            ]
          ]
        ],
        "type": "Polygon"
      }
    }
  ]
}

And the code:

    polygon = data["features"][0]["geometry"]
    coll = gd.MaskedCollection.from_name(collection)
    coll = coll.search(start_date="2019-01-01", end_date="2020-01-01", region=polygon, cloudless_portion=cloudless_portion)
    comp_im = coll.composite(method=mosaic, region=polygon)
    comp_im.download(output_path, region=polygon, crs='EPSG:32735', scale=10, max_tile_size=max_tile_size)

Here mosaic = 'medoid', cloudless_portion = 0.0 and for max_tile_size I've tried values of 8, 4, 1 and 0.1

[Vytautas Jancauskas]

Btw, if it helps, the error seems to happen before it starts downloading

[Dugal Harris]

OK. Can you tell me the value of collection?

[Vytautas Jancauskas]

'COPERNICUS/S2_SR_HARMONIZED'

[Dugal Harris]

The latest v1.7.0 release contains some improvements and an example related to this.