hatzav-ox / nwp-consumer_poetry

Microservice for consuming NWP data.

NWP CONSUMER

Microservice for consuming NWP data.

A microservice for consuming NWP data from various sources and storing it in a common format. Built with inspiration from the Hexagonal Architecture pattern, the nwp-consumer is currently packaged with adapters for pulling and converting .grib data from the MetOffice and from CEDA. Its modular nature enables straightforward extension to alternate future sources.

Running the service

Depending on the source and sink you choose to read and write data from, environment variables will need to be set. The program will inform you of missing env vars, but you can also check the config for the given module.

Using Docker (recommended)

This service is designed to be run as a Docker container. The Containerfile is the Dockerfile for the service. It is recommended to run it this way due to the present dependency on various external binaries, which at the moment cannot be easily distributed in a PyPi package. To run, pull the latest version from ghcr.io via:

$ docker run \
  -v /path/to/datadir:/data \
  -e ENV_VAR=<value> \
  ghcr.io/openclimatefix/nwp-consumer:latest <command...>  

Using the Python Package (not recommended)

Ensure the external dependencies are installed. Then, do one of the following:

• Download the latest wheel from the artifacts of the desired CI run and install it via

  $ pip install nwp-consumer-<version>.whl

• Clone the repository and install the package via

  $ pip install .

Then run the service via

$ ENV_VAR=<value> nwp-consumer <command...> 

Repository structure

Produced using exa:

$ exa --tree --git-ignore -F -I "*init*|test*.*"

./
├── Containerfile # The Dockerfile for the service
├── pyproject.toml # The build configuration for the service
├── README.md
└── src/
   ├── nwp_consumer/ # The main library package
   │  ├── cmd/
   │  │  └── main.py # The entrypoint to the service
   │  └── internal/ # Packages internal to the service. Like the 'lib' folder
   │     ├── config/ 
   │     │  └── config.py # Contains the configuration specification for running the service
   │     ├── inputs/ # Holds subpackages for each incoming data source
   │     │  ├── ceda/
   │     │  │  ├── _models.py
   │     │  │  ├── client.py # Contains the client and functions to map CEDA data to the service model
   │     │  │  └── README.md # Info about the CEDA data source
   │     │  └── metoffice/
   │     │     ├── _models.py
   │     │     ├── client.py # # Contains the client and functions to map MetOffice data to the service model
   │     │     └── README.md # Info about the MetOffice data source
   │     ├── models.py # Describes the internal data models for the service
   │     ├── outputs/ # Holds subpackages for each data sink
   │     │  ├── localfs/
   │     │  │  └── client.py # Contains the client for storing data on the local filesystem
   │     │  └── s3/
   │     │     └── client.py # Contains the client for storing data on S3
   │     └── service/ # Contains the business logic and use-cases of the application
   │        └── service.py # Defines the service class for the application, whose methods are the use-cases
   └── test_integration/

nwp-consumer is structured following principles from the hexagonal architecture pattern. In brief, this means a clear separation between the application's business logic - it's Core - and the Actors that are external to it. In this package, the core of the service is in internal/service/ and the actors are in internal/inputs/ and internal/outputs/. The service logic has no knowledge of the external actors, instead defining interfaces that the actors must implement. These are found in internal/models.py. The actors are then responsible for implementing these interfaces, and are dependency-injected in at runtime. This allows the service to be easily tested and extended. See further reading for more information.

Local development

Clone the repository and create and activate a new python virtualenv for it. cd to the repository root.

Install the External and Python dependencies as shown in the sections below.

External dependencies

The cfgrib python library depends on the ECMWF cfgrib binary, which is a wrapper around the ECMWF ecCodes library. One of these must be installed on the system and accessible as a shared library.

On a MacOS with HomeBrew use

$ brew install eccodes

Or if you manage binary packages with Conda use

$ conda install -c conda-forge cfgrib

As an alternative you may install the official source distribution by following the instructions at https://confluence.ecmwf.int/display/ECC/ecCodes+installation

You may run a simple selfcheck command to ensure that your system is set up correctly:

$ python -m <eccodes OR cfgrib> selfcheck
Found: ecCodes v2.27.0.
Your system is ready.

Python requirements

Install the required python dependencies and make it editable with

$ pip install -e . 

This looks for requirements specified in the pyproject.toml file.

Where is the requirements.txt file?

There is no requirements.txt file. Instead, the project uses setuptool's pyproject.toml integration to specify dependencies. This is a new feature of setuptools and pip, and is the recommended way to specify dependencies. See the setuptools guide and the PEP621 specification for more information, as well as Further Reading.

Running tests

Ensure you have installed the Python requirements and the External dependencies.

Run the unit tests with

$ python -m unittest discover -s src/nwp_consumer -p "test_*.py"

and the integration tests with

$ python -m unittest discover -s test_integration -p "test_*.py"

See further reading for more information on the src directory structure.

---

Further reading

On packaging a python project using setuptools and pyproject.toml:

• The official PyPA packaging guide.
• A step-by-step practical guide on the godatadriven blog.
• The pyproject.toml metadata specification.

On hexagonal architecture:

• A concrete example using Python.
• An overview of the fundamentals incorporating Typescript
• Another example using Go.

On the directory structure:

• The official PyPA discussion on src and flat layouts.

---

Contributing and community

• See the OCF Organisation Repo for details on contributing.
• Find out more about OCF in the Meta Repo.
• Follow OCF on Twitter.
• Check out the OCF blog at https://openclimatefix.org/blog for updates.