Giters

wmo-im / csv2bufr

Python package to transform CSV data into the WMO BUFR format

Home Page:https://csv2bufr.readthedocs.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

BUFR comparison

chenxiaoxia2019 opened this issue · comments

commented

Background

  • Now, there are two csv2bufr converters, one designed by Campbell, the other by David
  • source data is Malawi observation data retrieved from AWS server in .csv
  • output data is in BUFR format

Objective

  • use BUFR tools (ECWMF) to compare the two bufr files

Task

  • using bufr_compare to compare BUFR messages contained in two files. If some differences are found it fails to return an error code. Floating-point values are compared exactly by default, different tolerance can be defined see -P -A -R. Default behaviour: absolute error=0, bit-by-bit compare, the same order in files.
  • document the differences
  • fix the problems
commented

current differences from bufr_compare below. Campbell appear to be setting some missing data to 0 or the observed data are not included in the test files.

root@1ae6c5a3fd1b:/app# /opt/eccodes/bin/bufr_compare -A 0.001 ./data/output/0-454-2-awsnamitambo_2021-11-18_0955.bufr ./data/output/74dc5f1c3410f33ddb5b80d71a549abe.bufr4
== 1 == DIFFERENCE == long [internationalDataSubCategory]: [0] != [6]
== 1 == DIFFERENCE == long [dataSubCategory]: [0] != [110]
== 1 == DIFFERENCE == long [masterTablesVersionNumber]: [32] != [36]
== 1 == DIFFERENCE == long [typicalHour]: [10] != [9]
== 1 == DIFFERENCE == long [typicalMinute]: [0] != [55]
== 1 == DIFFERENCE == long [typicalSecond]: [11] != [0]
== 1 == DIFFERENCE == string [typicalTime]: [100011] != [095500]
== 1 == DIFFERENCE == string [wigosLocalIdentifierCharacter]: [awsnamitambo] != [AWSNAMITAMBO]
== 1 == DIFFERENCE == double [longitude]: [3.52700000000000031264e+01] != [3.52742800000000045202e+01]
        absolute diff. = 0.00428, relative diff. = 0.000121335
        tolerance=0.001
== 1 == DIFFERENCE == double [24HourPressureChange]: [0.00000000000000000000e+00] != [MISSING]
== 1 == DIFFERENCE == double [pressure]: [0.00000000000000000000e+00] != [MISSING]
== 1 == DIFFERENCE == long [nonCoordinateGeopotentialHeight]: [0] != [2147483647]
== 1 == DIFFERENCE == long [#2#totalSunshine]: [103] != [102]
== 1 == DIFFERENCE == long [#5#timePeriod]: [-1] != [2147483647]
== 1 == DIFFERENCE == double [#2#totalPrecipitationOrTotalWaterEquivalent]: [0.00000000000000000000e+00] != [MISSING]
== 1 == DIFFERENCE == long [#11#timePeriod]: [-10] != [2147483647]
== 1 == DIFFERENCE == double [#1#maximumWindGustSpeed]: [2.90000000000000035527e+00] != [5.70000000000000017764e+00]
        absolute diff. = 2.8, relative diff. = 0.491228
        tolerance=0.001
== 1 == DIFFERENCE == long [#12#timePeriod]: [-10] != [2147483647]
== 1 == DIFFERENCE == double [#2#maximumWindGustSpeed]: [2.90000000000000035527e+00] != [MISSING]
== 1 == DIFFERENCE == long [#13#timePeriod]: [0] != [2147483647]
== 1 == DIFFERENCE == long [typeOfInstrumentationForEvaporationMeasurement]: [0] != [2147483647]
== 1 == DIFFERENCE == double [evaporation]: [0.00000000000000000000e+00] != [MISSING]