Implement support for multiple releases per input file

[Stefan Rüdisühli]

• PyFlexPlot version: v0.15.3

Background

So far, input files always contained only one release from one release site (albeit with multiple species). To reduce the number of flexpart runs in operations, releases from multiple sites should be run in one flexpart run, which would substantially reduce the overall IO overhead from reading the meteorological field (which are identical for the individual sites).

Status

In principle, support for all dimensions present in the input files should be implemented, some were just always of size 1. If those are longer, some NotImplementedErrors etc. are likely to be triggered.

Tasks

• Assess what needs to be done
• Implement support for multiple releases
  • Basic support using "numpoint"/"noutrel"
  • Cleaner interface with "release_site" setup param
• Add tests (input/plot)

[Stefan Rüdisühli]

Assess situation

• Test case: /scratch/kaufmann/flexpart//job/1032/grid_conc/20210408000000.nc (6 releases)

Test dry run

• Run w/ --dry-run, w/o constrains on noutrel/numpoint: Works!
  • Creates six plots for each of the six release sites
  • The release sites are part of the output name (preset test/cosmo-1e-ctrl/concentration)
  • For the relases, numbered output names are derived (foo.png, foo.1.png, ...)
• Rerun with restricted noutrel and numpoint: Works!

Test real run

• Run w/o --dry-run: Exception raised in InputFileEnsemble._read_fld_over_time
  • Problem: Dimension "numpoint" not among dimensions of variable "spec001"

Structure of input file

• Dimension "numpoint": Multiple release points
• Dimension "noutrel": Multiple releases
• Only "noutrel" is among the dimensions of variable "spec001" etc.
  • The concentration and deposition fields only know which release they originate from, but not which release site
• The input file contains six release sites ("numpoint") and six releases ("noutrel"), one per site

Way forward

• Quick'n'dirty implementation in which "noutrel" must be specified for the desired results
• Add "release_site" setup param (index and/or name) from which the appropriate "noutrel" and "numpoint" indices are derived

[Stefan Rüdisühli]

Quick'n'dirty implementation

• numpoint isn't actually a dimension in the sense in which they are used within pyflexplot (i.e., an input field dimension)
  • Instead, it is merely a dimension in the NetCDF files because of the various variables describing the release point(s) (name, position, height, released amount etc.)
  • So it should really be removed as a from [Core]Dimensions!

Remove numpoint

• Go through the code and tests and remove numpoint from [Core]Dimensions
  • Problem: Input tests that read from input files fail because those do seem to contain numpoint!
• Check the input files of the different models:
  • COSMO-1, COSMO-1E, COSMO-1E-CTRL, COSMO-2E:
    • In file: numpoint, noutrel
    • In spec001: noutrel
    • Caution: In one COCSMO-1 file, noutrel is missing and numpoint is in spec001 instead...
  • COSMO-2, COSMO-E:
    • In file: numpoint
    • In spec001: numpoint
  • IFS-HRES, IFS-HRES-EU:
    • In file: pointspec, numpoint
    • In spec001: pointspec
• So which dimension is used differs widely and is not even guaranteed to be consistent for a given model type!
  • This calls for a flexible approach that checks the 2nd dimension of spec001
  • Also, the respective dimension in [Core]Dimensions should be abstracted, not one of numpoint/noutrel/pointspec

Merge noutrel and numpoint

• Only one of noutrel and numpoint is required by [Core]Dimensions
• However, they are used interchangeably in the input files, to a more abstract/neutral name is in order
• Let's go with "release"
• Implement this and test it: Works!
• Rerun the pyflexplot-test bash script with the range of test cases on tsa (dev vs. v0.15.4-pre): Identical!
• Merge into v0.15.4 and notify Pirmin
• Found a problem: Muehleberg is in the wrong place (near Fessenheim instead of near Bern)!
  • Looks like a problem with the input file, but ask Pirmin for confirmation