csv-validator

This project outlines an attempt to define a way of thinking about validating data in tabular format (CSV) inspired from its Python homologue Great Expectations. It's been optimized for performance and flexibility. It works by defining a set of validations that can be repeatedly applied to a tabular dataset in CSV format. This is especially useful in a datapipeline scenario where similar tabular (CSV) data is collected from different sources, and you like to validate the data before it enters downstream processing. The validations are stored in a validation suite that is written entirely in Common Lisp syntax.

Features:

Performant: single core validation speed is ~10MB/s and multicore validation speed is ~30MB/s. Please see Benchmark.
Extensible: validations are written in Common Lisp and stored in validation suites. New validations can be easily added.
No-nonsense output: output is a CSV file that contains location of values that did not pass the validation, including the erronuous value.
Well tested: each validation is unit-tested and compiles are tested to work on SBCL, ECL, CLISP, CCL and ABCL on Linux OS.

Table of Contents

Installation
Validating a simple CSV
Header validations
Record validations
Benchmark
Contributing

Installation

This package has been added to quicklisp. You can install it as follows:

Open the LISP REPL and run:
```
(ql:quickload 'csv-validator)
```
If the package doesn't seem available, please update your quicklisp distribution to the latest version with: (ql:update-dist "quicklisp").
Test if installation was succesfull by running:
```
(csv-validator:check-not-null "test-value")
```
If this returns T the installation was successful.

Validating a simple CSV

In the data/ folder the file energy_sample.csv is located. This small csv file is used to illustrate how the csv-validator works.

Let's start with defining a small validation suite and using it to perform a validation on the energy_sample.csv data. Open a new common lisp script and make sure that the csv-validator is correctly installed. Then define and run:

(defparameter *test_suite*
 (list
  (list
   :column "ID"                       ;The column to be validated
   :depends (list "ID")               ;The value to use in the logic (see below)
   :label "max-5-chars"               ;The text to include in output in case of failed validation
   :logic (lambda (ID)                ;The logic for this validation.
           (<= (length ID) 5)))))     ;The value in the ID column should have 5 or less characters.

(csv-validator:validate-csv "/path/to/data/energy_sample.csv"
"/path/to/output/folder/" *test_suite* :delim #\; :threads 1)

Make sure that you include the correct path to the energy_sample.csv input data and that the output folder that you're writing to exists. Now open the file named csv-validator_validations.csv in the output folder and explore the result:

index;column;erronuous_value;label
11;ID;999999;max-5-chars

As you can see the result itself is a CSV file (';' as delimiter). It correctly points out that the value in the ID column at index 11 is longer than 5 characters.

Header validations

Header validations are checks if the header row of the CSV file is as expected. This is especially usefull for data pipelines where the presence of certain headers in the CSV file is expected. Header validation will be performed for every validation in your validation suite. Example:

(defparameter *test_suite*
 (list
  (list
   :column "ID"
   :depends (list "ID")
   :label "max-5-chars"
   :logic (lambda (ID)
           (<= (length ID) 5)))
  (list
   :column "not_existing_column"
   :depends (list "not_existing_column")
   :label "max-10-chars"
   :logic (lambda (x)
           (<= (length x) 10)))))

Here a second validation is added to the validation suite that describes a column that doesn't exist in the data. This will give the following result after running the validation (through csv-validator:validate-csv ...:

index;column;erronuous_value;label
0;not_existing_column;not_existing_column;missing-header
11;ID;999999;max-5-chars

At index 0 (header row) the error with label "missing-header" shows up, because this header is not present in the CSV file. Since the ID column is present, we get the regular output for that column.

Record validations

Record validations are validations that are performed on each record in the CSV file. The csv-validator has several build-in validations, that are unit-tested. However, it's also perfectly possible to define your own functions or use lambda functions for validations.

Lambda validations

Literally every function can be used as a validation, including lambda functions. As long as the used function returns a non-nil value for correct values, and nil for incorrect values. For example:

(defparameter *test_suite*
 (list
   (list
     :column "country"
     :depends (list "country")
     :label "not-allowed-country"
     :logic (lambda (x) (position x '("AT" "BG" "CH") :test #'string=)))))

This lambda function validates if the country code is in the allowed list of country codes (in the file energy_sample.csv). The value from the column "country" will be used as x in the lambda function (as defined under :depends). For example the country code "ES" will result in an error in the result file like this:

index;column;erronuous_value;label
12;country;ES;not-allowed-country

Consider another example:

(defparameter *test_suite*
 (list
   (list
    :column "energy_source_level_0"
    :depends (list "energy_source_level_0")
    :label "true-or-false"
    :logic (lambda (x) (or (string= (string-downcase x) "true")
                           (string= (string-downcase x) "false"))))))

This lambda function validates the "energy_source_0" column and only allows for string-values "TRUE" or "FALSE", independent of case. The erronuous value "INVALID" in this column will result in:

index;column;erronuous_value;label
12;energy_source_level_0;INVALID;true-or-false

Conditional validations

Conditional validations are validations that are only performed on a subset of the column that is true for the condition. Using this concept, it's possible to define validations that are dependent on values in other columns in the same record. It's best illustrated using an example. In this example, the column "weblink" cannot have the value "link unavailable" when the source of the data is "REE":

(defparameter *test_suite*
 (list
  (list
   :column "weblink"
   :depends (list "weblink" "source")
   :label "unavailable-not-allowed"
   :logic (lambda (weblink source) (or (not (string= source "REE"))
                                       (not (string= (string-downcase weblink)
                                                     "link unavailable")))))))

This validation will only check if the value in the "weblink" column is "link unavailable" when the value in the "source" column is "REE". It'll give the following output upon running this suite against the energy_example.csv file:

index;column;erronuous_value;label
9;weblink;link unavailable;unavailable-not-allowed

Consider this other example:

(defparameter *test_suite*
 (list
  (list
   :column "capacity"
   :depends (list "capacity")
   :label "number"
   :logic (lambda (x) (or (csv-validator:check-null x)
                          (csv-validator:check-number-string x))))))

This validation validates if the values in the "capacity" column are numeric values, but only if these values are non-null values. Thus values like "null" or "3.14" will pass the validation, but a value like "ui44" won't (it's not a number). It uses several build-in validations that will be discussed in the sections below.

Numerical validations

Validate integers

Integer validation can be done using the build-in function: csv-validator:check-integer-string. It works as follows:

(csv-validator:check-integer-string "54") --> t
(csv-validator:check-integer-string "1.45") --> nil
(csv-validator:check-integer-string "-99") --> t
(csv-validator:check-integer-string "lk93") --> nil