dwd-csv-helper

dwd-csv-helper is a tiny helper package to extract timeseries data from .csv- and .kmz-files that have been downloaded by the microservice dwd_data_crawler.

The package is being developed and maintained by the Chair of Automation and Energy Systems at the Saarland University.

LICENSE

dwd-csv-helper is released under the ISC license.

Installation

$ npm install dwd-csv-helper

Dependencies and Compatibility

In addition to the dependencies defined in package.json, this software depends on the unzip command available on most Linux systems for extracting the .kml-file that gets parsed from the .kmz-file in which it gets delivered. This was introduced in commit b4863e5350cbffaf86c88002808f82cf414d2063.

Due to changes in the name and unit of some variables provided as part of the MOSMIX-forecast, the output of readTimeseriesDataMosmix was changed; this requires an update of dwd_data_access (compare the commit message)!

Usage

The dwd-csv-helper package exposes two functions.

readTimeseriesDataReport

Purpose: asynchronously extract a timeseries of measurement data for a given station id within a given time interval
Arguments:
1. (String): path to the directory comprising the directories for the individual days of measurement data (e.g. 'SOME_PATH/weather/weather_reports/poi')
2. (Number): inclusive start timestamp for the timeseries to be extracted as UNIX EPOCH in ms resolution (e.g. 1529280000000 for 2018-06-18 00:00 UTC)
3. (Number): exclusive end timestamp for the timeseries to be extracted as UNIX EPOCH in ms resolution (e.g. 1529366400000 for 2018-06-19 00:00 UTC)
4. (String): the identifier of the weather station for which to extract the timeseries (e.g. '10708' for weather station in Saarbrücken)
Returns (Object): a map of all timeseries items extracted.
- The keys of the Object refer to the values in the first line of the csv files
- The value behind the key is an Array of Object with the following attributes:
  - timestamp: the timestamp of the measurement value as UNIX EPOCH in ms resolution
  - value: the measurement value as raw value (no unit conversion)

const {
  readTimeseriesDataReport
} = require('dwd-csv-helper')

async function main () {

  // load and extrat timeseries
  let tsCollection
  try {
    tsCollection = await readTimeseriesDataReport(
      '/mnt/data/weather/weather_reports/poi', // must be adopted to the correct path
      1529280000000,                           // inclusive start timestamp: 2018-06-18 00:00 UTC
      1529366400000,                           // exclusive end timestamp: 2018-06-19 00:00 UTC
      '10708'                                  // weather station Saarbrücken
    )
  } catch (error) {
    console.error('something went wrong while extracting the timeseries')
    console.error(error)
    return
  }

  // display extracted timeseries data
  const keys = Object.keys(ts)
  for (let i = 0; i < keys.length; i++) {
    const timeseries = tsCollection[keys[i]]
    console.log(keys[i], timeseries)
  }
}

main()

readTimeseriesDataMosmix

Purpose:
- Asynchronously extract a timeseries of forecast data for a given reference time for a given station ID
- Take care of the fact that the format in which the MOSMIX-forecasts are provided by DWD changed on 2018-09-17.
Arguments:
1. (String): path to the directory comprising the directories for the individual forecasts (e.g. 'SOME_PATH/weather/local_forecasts/poi')
2. (Number): the reference timestamp of the forecast timeseries to be extracted as UNIX EPOCH in ms resolution (e.g. 1529301600000 for 2018-06-18 06:00 UTC)
3. (String): the identifier of the weather station for which to extract the timeseries (e.g. '10708' for weather station in Saarbrücken)
Returns (Object): a map of all timeseries items extracted
- The keys of the Object refer to the values in the first line of the csv files
- The value behind the key is an Array of Object with the following attributes:
  - timestamp: the timestamp of the measurement value as UNIX EPOCH in ms resolution
  - value: the measurement value as raw value (no unit conversion)

const {
  readTimeseriesDataReport
} = require('dwd-csv-helper')

async function main () {

  // load and extract timeseries data for a certain forecast
  let tsCollection
  try {
    tsCollection = await readTimeseriesDataMosmix(
      '/mnt/data/weather/local_forecasts/poi', // must be adopted to the correct path
      1529301600000,                           // inclusive start timestamp: 2018-06-18 06:00 UTC
      '10708'                                  // weather station Saarbrücken
    )
  } catch (error) {
    console.error('something went wrong while extracting the timeseries')
    console.error(error)
    return
  }

  // display extracted timeseries data
  const keys = Object.keys(ts)
  for (let i = 0; i < keys.length; i++) {
    const timeseries = tsCollection[keys[i]]
    console.log(keys[i], timeseries)
  }
}

main()

knut0815 / dwd-csv-helper