Giters

EduardRosert / regrid

A collection of tools and configs to interpolate/regrid DWD's ICON data from triangular/icosahedral to regular latitude-longiture/geographical grid.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

CDO version Python version ecCodes version zlib version HDF5 version NetCDF version JasPer version Docker Build Status Docker Pulls

Regrid DWD ICON Data

A collection of tools and configurations that allow you to interpolate/regrid DWD's NWP ICON model data from icosahedral(triangular) ICON grids onto regular latitude-longitude (geographical) grids using MPIMET CDO and ECMWF ecCodes. DWD's NWP data is freely available for download from DWD's Open Data File Server at https://opendata.dwd.de/weather/nwp/ .

Table of Contents

Table of contents generated with markdown-toc

Available images

There are a variety of different docker images available for different regridding tasks.

Model specific images

Docker images that are tagged with a model name such as icon-eu-eps or all provide all necessary files to regrid data from these models. Currently the following models are supported:

  • ICON Global: deutscherwetterdienst/regrid:icon
  • ICON Global Ensemble: deutscherwetterdienst/regrid:icon-eps
  • ICON EU Nest Ensemble: deutscherwetterdienst/regrid:icon-eu-eps
  • ICON-D2 (pre-operational): deutscherwetterdienst/regrid:icon-d2
  • ICON-D2 Ensemble (pre-operational): deutscherwetterdienst/regrid:icon-d2-eps

Docker Tags

The general nomenclature for tags is as follows:

deutscherwetterdienst/regrid:[<model>|all][-samples|-grids][<version>|-<version>]

<model>

These images contain precomputed interpolation weights located in /data/weights/ for remapping model data to a regular-lat-lon/geographical grid at full resolution, e.g.:

  • ICON EU Nest Ensemble: deutscherwetterdienst/regrid:icon-eu-eps
  • all supported models: deutscherwetterdienst/regrid:all

-samples

These images contain GRIB2 samples files for the given models e.g.

  • image with sample data for all models: deutscherwetterdienst/regrid:all-samples
  • image with ICON global samples: deutscherwetterdienst/regrid:icon-samples Samples are located in /data/samples/.

-grids

These images contain NetCDF grid definition files. These files are necessary to create customized interpolations for the given model and are located in /data/grids/. Sample files are also included in /data/samples/ to simplify tests. Examples:

  • image with definition and sample files for all models: deutscherwetterdienst/regrid:all-grids
  • image with definition and sample files for icon model: deutscherwetterdienst/regrid:icon-grids

<version>|-<version>

Images tagged with a version suffix correspond to a specific version tag in the VCS and are considered immutable. Images without a version tag signify the latest available versions.

Images for advanced users

We provide a slim/minimal image that contain all precompiled tools such as ecCodes and CDO, which are required to regrid/interpolate GRIB2 and NetCDF files:

deutscherwetterdienst/regrid

These images are intended for advanced users with existing knowledge of CDO. They do not provide any model specific configurations or data and hence can not be used to regrid data without providing additional data and configurations.

Usage

You can use the provided images interpolate data onto predefined grids or fully customize all details of the interpolation.

Get Version Information

You can output versions the CDO version and the versions of all libraries by running:

docker run --rm deutscherwetterdienst/regrid cdo --version

The result should look like this:

Climate Data Operators version 1.9.8 (https://mpimet.mpg.de/cdo)
System: x86_64-pc-linux-gnu
CXX Compiler: g++ -g -O2 -fopenmp 
CXX version : g++ (Debian 8.3.0-6) 8.3.0
C Compiler: gcc -fPIC -fopenmp  
C version : gcc (Debian 8.3.0-6) 8.3.0
F77 Compiler: gfortran -g -O2
F77 version : GNU Fortran (Debian 8.3.0-6) 8.3.0
Features: 1GB 4threads C++14 Fortran DATA PTHREADS OpenMP HDF5 NC4/HDF5/threadsafe OPeNDAP SSE2
Libraries: HDF5/1.12.0
Filetypes: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5 
     CDI library version : 1.9.8
 cgribex library version : 1.9.4
 ecCodes library version : 2.17.1
  NetCDF library version : 4.7.4 of Jun 23 2020 18:51:27 $
    hdf5 library version : 1.12.0 threadsafe
    exse library version : 1.4.1
    FILE library version : 1.8.3

Basic interpolation

We provide tailor made images for our models that allow you to directly interpolate all available data from icosahedral/triangular ICON grids to regular-latitude-longitude/geographical grids with maximum resolution.

Regrid sample files

You can use images tagged with -samples to test interpolation on sample files. Run:

docker run --rm deutscherwetterdienst/regrid:icon-eu-eps-samples

This will convert the sample file, save the result in /data/samples/icon/icon_output.grib2 and exit. The output should look like this:

cdo    remap: Processed 40 variables over 1 timestep [0.32s 79MB].

Regrid custom files

You can customize which data should be interpolated and where the result should be written by overriding the following environment variables in the docker image using the --env <VARIABLE>=<VALUE> option of docker.

  • INPUT_FILE: Absolute file path to the input file in GRIB2 format that needs to be transformed, e.g. /data/samples/icon/icon_sample.grib2
  • OUTPUT_FILE: Absulute path where the result should be written, e.g.: /data/samples/icon/icon_output.grib2

Example 1: Regrid a single file from you local hard drive

You need to mount a folder from you local hard drive and specify which file to regrid and where to save the output by setting the environment variables accordingly:

docker run --rm \
    --volume ~/mydata:/mydata \
    --env INPUT_FILE=/mydata/my_icon-eps_icosahedral_file.grib2 \
    --env OUTPUT_FILE=/mydata/regridded_regular_lat_lon_output.grib2 \
    deutscherwetterdienst/regrid:icon-eps

The output should look something like this:

cdo    remap: Processed 40 variables over 1 timestep [0.32s 79MB].

Attention: The model of the docker image and the grid of the selected file have match!

Example 2: Interpolate all files from a local folder

To simplify the batch interpolation of files, the script convert.sh is included in every image. You can use environment variables or command line options to control the script's behaviour:

Regrids grib2 files and writes the output to the output directory. 

Usage: convert.sh [-i <INPUT_FILE>] [-o <OUTPUT_FILE>] [-w <WEIGHTS_FILE>] [-d <DESCRIPTION_FILE>] 

 
Alternatively the following environment variables can be used to control this script: 
	 - INPUT_FILE : grib2 file or directory containing grib file(s) 
	 - OUTPUT_FILE : grib2 file or directory to output regridded file(s) 
	 - WEIGHTS_FILE : netCDF file that contains interpolation weights 
	 - DESCRIPTION_FILE : cdo target grid description file for the given weights file 

 
Example (with command line options): 
	 convert.sh \ 
		 -i /data/samples/icon/ \ 
		 -o /output/ \ 
		 -w /data/weights/icon/icon_weights.nc \ 
		 -d /data/descriptions/icon/icon_description.txt 
 
Example (with environment variables): 
	 export INPUT_FILE=/data/samples/icon/ 
	 export OUTPUT_FILE=/output/ 
	 export WEIGHTS_FILE=/data/weights/icon/icon_weights.nc 
	 export DESCRIPTION_FILE=/data/descriptions/icon/icon_description.txt 
	 convert.sh
  1. Place all GRIB2 files that need to converted into a single folder, e.g ~/mydata. It's important that all these files are from the same model, e.g. the model icon-d2-eps. Now the contents of you folder should look something like this:
eduard@Eduards-MacBook-Air mydata % ls -la
total 262080
drwxr-xr-x    8 eduard  staff       256 Jul 29 19:44 .
drwxr-xr-x@ 127 eduard  staff      4064 Jul 29 19:38 ..
-rw-r--r--    1 eduard  staff  22361460 Jul 29 19:44 icon-d2-eps_germany_icosahedral_single-level_2020072900_000_2d_t_2m.grib2
-rw-r--r--    1 eduard  staff  22361460 Jul 29 19:44 icon-d2-eps_germany_icosahedral_single-level_2020072900_001_2d_t_2m.grib2
-rw-r--r--    1 eduard  staff  22361460 Jul 29 19:44 icon-d2-eps_germany_icosahedral_single-level_2020072900_002_2d_t_2m.grib2
...
...
  1. Create a folder that will hold regridded data, e.g. ~/output.
mkdir ~/output
  1. Mount the folders inside your container (make sure you use absolute paths) and run the convert.sh script located in file system root. Make sure you selected an image that matches the data you want to regrid, e.g. deutscherwetterdienst/regrid:icon-d2-eps for icon-d2-eps data:
docker run --rm \
    --volume ~/mydata:/mydata \
    --volume ~/output:/output \
    --env INPUT_FILE=/mydata \
    --env OUTPUT_FILE=/output \
    deutscherwetterdienst/regrid:icon-d2-eps \
    /convert.sh

The output should look something like this:

Regridding all files in directory '/mydata' ...
Regridding '/mydata/icon-d2-eps_germany_icosahedral_single-level_2020072900_000_2d_t_2m.grib2' > '/output/regridded_icon-d2-eps_germany_icosahedral_single-level_2020072900_000_2d_t_2m.grib2'...
cdo    remap: Processed 20 variables over 1 timestep [1.01s 170MB].
Regridding '/mydata/icon-d2-eps_germany_icosahedral_single-level_2020072900_001_2d_t_2m.grib2' > '/output/regridded_icon-d2-eps_germany_icosahedral_single-level_2020072900_001_2d_t_2m.grib2'...
cdo    remap: Processed 20 variables over 1 timestep [0.93s 170MB].
Regridding '/mydata/icon-d2-eps_germany_icosahedral_single-level_2020072900_002_2d_t_2m.grib2' > '/output/regridded_icon-d2-eps_germany_icosahedral_single-level_2020072900_002_2d_t_2m.grib2'...
cdo    remap: Processed 20 variables over 1 timestep [1.04s 170MB].

...

Done.

Your folder ~/output should now contain the regridded files:

eduard@Eduards-MacBook-Air output % ls -la
total 621800
drwxr-xr-x   20 eduard  staff       640 Jul 29 19:45 .
drwxr-xr-x@ 127 eduard  staff      4064 Jul 29 19:38 ..
-rw-r--r--    1 eduard  staff  32220340 Jul 29 19:44 regridded_icon-d2-eps_germany_icosahedral_single-level_2020072900_000_2d_t_2m.grib2
-rw-r--r--    1 eduard  staff  32220340 Jul 29 19:44 regridded_icon-d2-eps_germany_icosahedral_single-level_2020072900_001_2d_t_2m.grib2
-rw-r--r--    1 eduard  staff  32220340 Jul 29 19:44 regridded_icon-d2-eps_germany_icosahedral_single-level_2020072900_002_2d_t_2m.grib2
...

Interactive usage

You can use all images by running command direcrly inside the container using an interactive shell.

Interactive shell

Run:

docker run --rm -it \
    deutscherwetterdienst/regrid:icon-samples \
    bash

Analyse input

Analyse the input file by running:

grib_ls /data/samples/icon/icon_sample.grib2

The output should look something like this:

/data/samples/icon/icon_sample.grib2
edition      centre       date         dataType     gridType     stepRange    typeOfLevel  level        shortName    packingType  
2            edzw         20200618     fc           unstructured_grid  0            heightAboveGround  2            2t           grid_simple 
1 of 1 messages in /data/samples/icon/icon_sample.grib2

1 of 1 total messages in 1 files

Interpolate from triangular to regular-lat-lon grid

Run:

cdo -f grb2 remap,/data/descriptions/icon/icon_description.txt,/data/weights/icon/icon_weights.nc /data/samples/icon/icon_sample.grib2 /data/samples/icon/icon_regridded_output.grib2

Analyse output

To analyse the output file run:

grib_ls /data/samples/icon/icon_regridded_output.grib2

The result should look something like this:

/data/samples/icon/icon_regridded_output.grib2
edition      centre       date         dataType     gridType     stepRange    typeOfLevel  level        shortName    packingType  
2            edzw         20200618     fc           regular_ll   0            heightAboveGround  2            2t           grid_simple 
1 of 1 messages in /data/samples/icon/icon_regridded_output.grib2

1 of 1 total messages in 1 files

The gridType changed from unstructured_grid (ICON triangular grid) to regular_ll (geographical grid).

Custom regridding

You can fully customize which and how data will be interpolated by modifying environment variables and providing custom grid description files and weights files.

Environment Variables

The following environment variables are supported:

  • INPUT_FILE: Absolute file path to the input file in GRIB2 format that needs to be transformed, e.g. /data/samples/icon/icon_sample.grib2
  • OUTPUT_FILE: Absulute path where the result should be written, e.g. /data/samples/icon/icon_output.grib2
  • DESCRIPTION_FILE: The CDO grid description file, e.g. /data/descriptions/icon/icon_description.txt
  • WEIGHTS_FILE: pre-computed interpolation weights - see DWD's CDO How-To-Guide, e.g. /data/weights/icon/icon_weights.nc
  • GRID_FILE: NetCDF grid definition file, e.g. /data/grids/icon/icon_grid.nc

Custom grid descriptions and weights

You can fully customize how to regrid DWD grib data by first creating a CDO grid description file and then genrating the interpolation weights. It is possible to extract an arbitrary subset of the available data and/or interpolate the data onto a custom grid.

Example 1: Create a CDO grid description file

First you need to create a custom (CDO grid description file)[https://code.mpimet.mpg.de/projects/cdo/embedded/index.html#x1-150001.3.2]. This file defines the grid type, the area and the resolution of the desired grib file after interpolation. The file below is an example of a cutout over Europe in a regular latitude-longitude (geographical) grid with a resolution of 0.25° x 0.25° degrees:

# File: output_grid.txt
# Climate Data Operator (CDO) grid description file
# Input: ICON (Global)
# Area: Europe
# Grid: regular latitude longitude/geographical grid
# Resolution: 0.25 x 0.25 degrees

gridtype  = lonlat
xsize     = 601
ysize     = 301
xfirst    = -75.0
xinc      = 0.25
yfirst    = 5.0
yinc      = 0.25

Save this file to a local folder, e.g. to ~/mydata/custom_grid.txt.

Example 2: Generate weights and regrid

In order to generate weights, you need the grid definition files (in NetCDF) that are necessary to generate custom interpolation weights for the icon model you want to interpolate. To do this we will use the corresponding grids images. In this example we will use the ICON Global image tagged icon-grids, which we will run interactively:

docker run --rm -it \
    --volume ~/mydata:/mydata \
    --workdir /mydata \
    deutscherwetterdienst/regrid:icon-grids \
    bash

Now, to precompute the weights for the custom grid definition run the following command inside the container:

cdo gennn,/mydata/custom_grid.txt \
    /data/grids/icon/icon_grid.nc \
    /mydata/custom_weights.nc

Attention: For ICON-D2 (and ICON-D2-EPS) you need to use the -setgrid option and select grid no. 2 in the netCDF grid definition file to generate weights:

cdo gennn,/mydata/custom_grid.txt \
  -setgrid,/data/grids/icon-d2/icon-d2_grid.nc:2 \
  /data/samples/icon-d2/icon-d2_sample.grib2 \
  /mydata/custom_weights.nc

The output should look something like this:

cdo    gennn: Nearest neighbor weights from unstructured (2949120) to lonlat (601x301) grid
cdo    gennn: Processed 6 variables over 1 timestep [22.84s 932MB].

Now you can use the newly generated custom weights file custom_weights.nc to regrid the icon sample file:

cdo -f grb2 remap,/mydata/custom_grid.txt,/mydata/custom_weights.nc /data/samples/icon/icon_sample.grib2 /mydata/icon_custom.grib2

About

A collection of tools and configs to interpolate/regrid DWD's ICON data from triangular/icosahedral to regular latitude-longiture/geographical grid.

License:Apache License 2.0

Languages

Language:Dockerfile 70.9%Language:Shell 26.2%Language:Makefile 2.9%