disdroDL is a Python software for logging data from the OTT Parsivel2 and Thies Clima optical disdrometers and export it to 24-hours NetCDF files. It was developed at TU Delft, within the framework of the Ruisdael Observatory.
Parsivel2 disdrometer in the Cabauw tower, Netherlands.
One of the key aspects in disdroDL is the decision to separate code logic from the NetCDF structure and metadata. During the creation of the NetCDFs, a Parsivel general yaml file or a Thies general yaml file containing the description of Parsivel/Thies telegram variables and dimensions, that is applicable to all the Parsivel/Thies devices; is combined with site-specific metadata files that describe the variable components of the metadata such as location, name, etc.
The software features a main script (main.py) for setting up a serial connection with the disdrometers, requesting data at regular time intervals, and storing the Telegram data in a local sqlite3 database file. And an export script (export_disdrodlDB2NC.py) that exports 1 day of disdrometer data, from the database onto a NetCDF file.
What data is included in the NetCDF depends on the configuration files and whether the exported netCDF is a light or full version (described in Outputs). The NetCDF files are self-descriptive, and include metadata information about dimensions, variables names and units.
The structure of the NetCDF file depends on the sensor type and two configuration files, a general and site-specific one. The general configuration files configs_netcdf/config_general_parsivel.yml and configs_netcdf/config_general_thies.yml are applicable to all sensors of the same type, while the specific configuration files, 1 file per sensor (in configs_netcdf/), describe the variable components such as site names, coordinates, etc.
The Parsivel2 measures the drop number concentrations for different diameter/velocity bins, with a temporal resolution of 1 minute. These raw spectra can be used to calculate many different state and flux variables, such as liquid water content, rainfall rate, mean drop diameter, radar reflectivity and kinetic energy.
- Time is set to UTC
- the data destination directory: defined in configs_netcdf/ site specific config files, under the
data_dirvariable - monthly data directories
yyyymmare created inside parent data directory - every day, a new NetCDF file is created, with the following naming convention:
yyyymmdd_{site_name}_{station_code}_{sensor_name}.nc
- operating system: linux-based
- install NetCDF utilities
sudo apt install netcdf-bin - create a python virtual environment (venv)
- installed python dependencies in venv:
pip -r requirements.txt - create a log directory with read and write permissions to all users:
sudo mkdir /var/log/disdroDL/; sudo chmod a+rw /var/log/disdroDL - create a data directory with read and write permissions to all users:
sudo mkdir /data/disdroDL/; sudo chmod a+rw /data/disdroDL - run reset_parsivel:
python reset_sensor.py -c config_*.ymlto reset the sensor time and accumulated rain amount (TODO:confirm) - create a station-specific file and commit it to this repo (see configs_netcdf/config_008_GV.yml as an example)
- install netcdf-bin:
sudo apt install netcdf-bin, to be able to compress NetCDFs
If you have run a previous version of disdroDL, you might need to update the database schema. To do this, run the following script:
python upgrade_db.py --config config_*.yml
Make sure you run this script with the same config file that was used to run the previous version of disdroDL.
- OTT Parsivel2 - https://www.ott.com/download/operating-instructions-present-weather-sensor-ott-parsivel2-with-screen-heating-3/
- Thies Clima LPM - https://www.thiesclima.com/db/dnl/5.4110.xx.x00_Laser_Precipitation_Monitor_eng.pdf
Manually:
-
Writes Parsivel/Thies Telegrams to sqlite3 DB:
python main.py --config configs_netcdf/config_008_GV.yml(usually runs as service, but can also be run as a standalone script) -
Export DB entries of one day to a NetCDF
python export_disdrodlDB2NC.py (--version light/full) --date 2023-12-24 --config configs_netcdf/config_008_GV.yml
As Linux Systemd Service:
- edit the service file disdrodl.service changing the config file it will use
- create system link between local service file and service files location:
ln disdrodl.service /etc/systemd/system/disdrodl.service - run:
systemctl enable disdrodl.service - run:
systemctl start disdrodl.service - check status:
systemctl status disdrodl.service
Light vs full netCDFs
- The software can output a light or full NetCDF. This can be done by selecting
--version lightor--version full. Which variables will be written to the light/full NetCDF depends on their include_in_nc filed in the configuration files. The field can be assigned values: 'always', 'only_full' or 'never'. Variables assigned 'always' will be included in both light and full NetCDFs. Variables assigned 'only_full' will be included only in full netCDFs. Variables assigned 'never' will not be included in either. *default: full NetCDF.
netCDF output
Note that some of the fields sent by the Parsivel are discarded during the creation of the NetCDF file. For example, all the 16bit fields are discarded and only the 32bit values are stored. Rainfall accumulation (field 24) is discarded because it is relative to an unknown starting time and can be re-calculated from the rain rate. Sensor time/date (fields 20-21) are replaced by the actual time (in UTC) of the computer running the logging software. This is more reliable than to use the internal clock of the Parsivel which can drift over time. Sample interval (field 9) is ignored, because it can be inferred from the time difference between successive measurements.
No quality control is applied to the output.
The NetCDF files are automatically compressed.
- Main script: main.py
- Configuration files:
- general Parsivel: configs_netcdf/config_general_parsivel.yml
- general Thies: configs_netcdf/config_general_thies.yml
- specific Parsivel: e.g., configs_netcdf/config_008_GV.yml - create 1 per Parsivel
- specific Thies: e.g., configs_netcdf/config_006_GV_THIES.yml - create 1 per Thies
- Export script export_disdrodlDB2NC.py - exports 1 day of measurements from DB to NetCDF file
- Functions and classes are in their matching files in the modules folder:
- function for creating the logger - modules/log.py
- NetCDF classes and functionality - modules/netCDF.py
- class for getting current time - modules/now_time.py
- sensor abstract class and Parsivel/Thies sensor classes - modules/sensors.py
- functions for communicating with the database - modules/sqldb.py
- telegram abstract class and Parsivel/Thies telegram classes - modules/telegram.py
- utility functions - modules/util_functions.py
main.py (often as service, see example disdrodl.service)
- reads configurations from configs_netcdf/config_general_parsivel.yml or configs_netcdf/config_general_thies.yml and target-device config
- sets up the serial communication with the Parsivel/Thies
- in a while loop (every minute):
- requests the telegram from OTT Parsivel2/Thies Clima, outputting all measurement values :
CS/PA<CR> - appends the received telegram data into
disdro.db
- requests the telegram from OTT Parsivel2/Thies Clima, outputting all measurement values :
- reads configurations from configs_netcdf/config_general_parsivel.yml or configs_netcdf/config_general_thies.yml and target-device config
- queries
disdro.dbfor entries between 00:00:00 and 23:59:59 of the date provided to arg--date - for each returned database entry:
- the telegram (db column) value is parsed in an instance of the
ParsivelTelegram/ThiesTelegram - instance of the
ParsivelTelegram/ThiesTelegramis appended totelegram_objslist
- the telegram (db column) value is parsed in an instance of the
telegram_objsis provided to an instance of theNetCDFclass, which- creates a NetCDF file
- writes the
telegram_objsdata into the NetCDF - compresses the NetCDF file using
nccopy -d9
disdrodl.service - Linux's systemd service file responsible for running main.py as a service
- requires editing: replace default path of config file, with config for the instrument in question.
Parser for historical Ruisdael's OTT Parsivel CSVs. Converts CSV to netCDF
For more information CONVERSIONS.md
Run: python parse_disdro_csv_or_txt.py -c configs_netcdf/config_007_CABAUW.yml -i sample_data/20231106_PAR007_CabauwTower.csv
run: pytest -s
Tests are run Github Actions. See .github/workflows/test_n_lint.yml
with 2 different screens (use tmux multiplexer or 2 different shells)
terminal one: listen to serial port tail -f /dev/ttyUSB0
terminal two: send commands to serial port echo -en "CS/L\r" > /dev/ttyUSB0
disdroDL is developed in the context of the Ruisdael Observatory by
- Marc Schleiss (Principal Investigator)
- Andre Castro
- Mahaut Sourzac
- Saverio Guzzo
- Rob MacKenzie
- Vasil Chirov
- Mels Lutgerink
- Ella Milinovic
- Noky Soekarman
- Jesse Vleeschdraager
GPLv3. See LICENSE