UM2NETCDF
Mark Cheeseman
National Institute of Water & Atmospheric Research Ltd (NIWA)
m.cheeseman@niwa.co.nz
-------------------------------------------------------------------------------
TABLE OF CONTENTS
1. Overview
2. How to build the um2netcdf executable
3. How to run
4. Important notes
1. OVERVIEW
-------------------------------------------------------------------------------
UM2NetCDF is an utility program that can convert a binary fieldsfile from the
UK MetOffice's Unified Model into a NetCDF file. Effort has been made to
ensure that the output NetCDF file is [almost fully** ] compliant with the current
CF rules (as of June 30, 2014).
UM2NetCDF offers the following features:
-> the utility can be used on UM fields or dump files.
-> users can output the entire contents of an input UM input file or select a
sub-set of variables by entering a list of UM stash codes on the commandline
-> users can select the output UM fields be written with 32 or 64 bit
precision
-> ZLIB compression is utilized in the output NetCDF file offering substantial
storage savings
-> 'generic' variables (Eg. UM variables possessing data slices undergoing
postprocessing) are automatically split into separate datafields that
are appropriately named based on the post-processing performed.
-> An UM variable that has undergone post-processing (eg. their LBPROC value
is non-zero) will have an appropriate prefix added to its name.
For example, a field collecting maximum wind gusts would be named 'max_wind_gust'.
-> the utility can extract and process UM variables that are unpacked or are
WGDOS packed for x86-Linux and AIX_IBM Power architectures.
2. BUILDING UM2NETCDF
-------------------------------------------------------------------------------
2.1 Prerequisites
To build UM2NetCDF, one needs the following available:
- a C compiler (GNU, PGI and IBM XLC compilers have been tested)
- XML2 library (versions 2.9.0 and 2.9.1 have been tested and are
recommended)
- NetCDF with the netcdf-4 option selected during its build (this means
that it was build with a version of HDF5) (Versions 4.3.0-4.3.2
have been tested and are recommended) Note that this should be a serial
NetCDF implementation (eg. no MPI-IO support).
2.2 Configuration
The subdirectory 'config' contains text files describing site and machine-
specific information required for a build. Currently, there are files
describing information related to x86 and IBM Power6 architecture-based
platforms at NIWA. GNU and PGI-based configurations for the x86 platforms
at NIWA are also available.
It is suggested that you modify one of these exisiting files with the
specifics of your particular site and/or platform.
2.3 Building the executable
The build process contains only 2 simple steps:
cd src
make ARCH=XXXX CC=YYYY
XXXX represents the platform on which you are building UM2NetCDF. This
can only be set to 'x86' or 'pwr6' currently. YYYY represents the compiler
you wish to use for the compliation and linking of the binary. Currently,
this can be set to 'gcc', 'pgcc' or 'xlc'.
Notes:
A) The Makefile will try to include/read the appropriate machine file
located in the 'config' subdirectory based on your ARCH and CC values.
They must be the same as those specified in the machine file you wish
to use. For example, if you want to build the UM2Netcdf binary on a
x86 machine with the GNU C compiler, you would use the make command:
make ARCH=x86 CC=gcc
The Makefile would look for the machine file called make.inc.x86_gcc
in the 'config' subdirectory.
B) It is assumed that you are assuming GNU make for the build. The IBM
make that comes by default with AIX will not work. Use the 'gmake'
command instead of 'make' in the above instructions.
C) Building the executable on a big-endian architecture, such as IBM Power6/7
requires the inclusion of a CPP flag (-DIBM_POWER).
3. RUNNING UM2NETCDF
-------------------------------------------------------------------------------
3.1 Run directory
By default, the um2netcdf.x binary will be built in the 'run' subdirectory.
You can change this setting in the Makefile located in the 'src'
subdirectory.
3.2 Environment setup
You may need to manually set the environment so that the NetCDF library you
are using is found. This setting procedure depends on the OS of the
platform in which you are building. For Linux, you would use the command:
export LD_LIBRARY_PATH=<path_to_NetCDF_lib_directory>:${LD_LIBRARY_PATH}
For AIX, you would use the command:
export LIBPATH=<path_to_NetCDF_lib_directory>:${LIBPATH}
3.3 Running
UM2NetCDF is an interactive command will the following structure:
./um2netcdf.x [FLAGS] [Input UM fields file] [XML Field Definitions File]
where
FLAGS There are 6 flags that users can specify in an um2netcdf run:
-h will display a help message outlining the various options that
can be used with UM2NetCDF
-i all fields (found in the input UM fields file or specified by
the user) are written on the thermodynamic grid (eg. P-points on
an Arakawa-C grid)
-o <FILENAME>
used for when the user wishes to specify a particular file for
the resulting output NetCDF file.
-r all fields written in the NetCDF file are written with 32-bit
precision. (Eg floats instead of doubles, integers instead of
longs)
-s STASH_CODE1 STASH_CODE2 ...
Flag allows users to specifiy a space delimited list of UM
variables to be extracted from the input UM fields file and
written into the NetCDF file. STASH codes provided must exist
in the XML field definition file used in the call to um2netcdf.x
-c <XML Run Configuration File>
Specifies the name of the XML file containing required run
configuration data. UM2NetCDF will not run if this file is not
specified. An example of such a file can be found in the 'run'
directory.
The Input UM fields file must contain unpacked data. It can be of little
or big endian format. Note that the name of the output NetCDF file will
be the same as the input UM fields file with the '.nc' suffix appended.
The XML field definitions file contains all the necessary CF metadata
required for any output NetCDF file to be CF-compliant. A copy of this
file can be found in 'run' subdirectory. Be careful to follow the XML
structure of the file when adding/modifying fields.
4. IMPORTANT NOTES
-------------------------------------------------------------------------------
-> UM2NetCDF only works on UM fields and dump files. PP and other files are
not supported.
-> UM2NetCDF uses a NetCDF-4 standard/version of that library. The resulting
output NetCDF file may be compatible for some existing post-processing
tools. XCONV will not recognize the output NetCDF files. NCVIEW and NCL
will recognize the output NetCDF files.
-> The ZLIB compression can lead to longer execution times particularly on
larger input UM fields files.
-> WGDOS unpacking is available for little-endian [x86] and big-endian [IBM Power]
machines.