Izaakwltn / rs-json

Yet another JSON decoder/encoder for Common Lisp

RS-JSON

Yet another JSON decoder/encoder.

If you can’t wait until YASON is fixed, then this library is for you. The main differences are listed below.

• The parser is strictly RFC 8259 compliant where it makes sense. However, you can tweak the behaviour of the parser to suite your needs.
• The serializer only supports a compact pretty printing format.
• JSON objects can be represented as hash-tables, associated lists, or property lists. The default is to use alists.
• JSON arrays can be represented as vectors or lists. The default is to use vectors.
• The JSON values true, false, and null are represented by the keywords :true, :false, and :null respectively. But you can change that to suite your needs.
• The default configuration is round-trip save, i.e. you can read a JSON value and write it back without loss of information. This is a strict requirement when updating a web resource via an HTTP GET/PUT cycle.
• Performance is competitive to other “fast” JSON libraries out there.

The Decoder

The parse function is the entry point for reading a JSON value as a Lisp data structure. For example,

(parse "[{\"foo\" : 42, \"bar\" : [\"baz\", \"hack\"]}, null]")

 ⇒ #((("foo" . 42) ("bar" . #("baz" "hack"))) :null)

There are many user options to control the behaviour of the parser. Please check the documentation.

The Encoder

The serialize function is the entry point for printing a Lisp data structure as a JSON value. For example,

(serialize nil #((("foo" . 42) ("bar" . #("baz" "hack"))) :null))

 ⇒ "[{\"foo\" : 42, \"bar\" : [\"baz\", \"hack\"]}, null]")

Pretty Printing

If the pretty printer is disabled, JSON output is just a flat sequence of characters. For example:

[{"foo" : 42, "bar" : ["baz", "hack"]}, null]

There is no explicit or implicit line break since all control characters are escaped. While this is fast and machine readable, it’s difficult for humans to reveal the structure of the data.

If the pretty printer is enabled, JSON output is more visually appearing. Here is the same example as above but pretty printed:

[{"foo" : 42,
  "bar" : ["baz",
           "hack"]},
 null]

Explicit line breaks occur after object members and array elements and the items of these compound structures are lined up nicely.

Formatted Output

JSON output via Common Lisp’s format function is fully supported. There are two options. The first option is the question mark ~? directive. For example,

(format t "JSON: ~@?~%" serializer #((("foo" . 42) ("bar" . #("baz" "hack"))) :null))

may print

JSON: [{"foo" : 42,
        "bar" : ["baz",
                 "hack"]},
       null]

The value of the serializer constant is a format control function that follows the conventions for a function created by the formatter macro and the value of the *print-pretty* special variable determines if the JSON output is pretty printed.

The second option is the serializer function. This function is designed so that it can be called by a slash format directive. For example,

(format t "JSON: ~/serializer/~%" #((("foo" . 42) ("bar" . #("baz" "hack"))) :null))

prints

JSON: [{"foo" : 42, "bar" : ["baz", "hack"]}, null]

If the colon modifier is given, use the pretty printer. For example,

(format t "JSON: ~:/serializer/~%" #((("foo" . 42) ("bar" . #("baz" "hack"))) :null))

prints

JSON: [{"foo" : 42,
        "bar" : ["baz",
                 "hack"]},
       null]

Please note that the indentation of the pretty printed JSON output is relative to the start column of the JSON output.

User Defined Data Types

The RS-JSON library provides means to encode/decode user defined data types. Here is an example for a CLOS class.

(defclass user ()
  ((name
    :initarg :name
    :initform (error "Missing user name argument."))
   (id
    :initarg :id
    :initform nil)))

For encoding, define an encode method for the class.

(defmethod encode ((object user))
  "Encode an user as a JSON object."
  (let ((*encode-symbol-hook* :downcase))
    (with-object
      (object-member "" (type-of object))
      (iter (for slot :in '(name id))
            (object-member slot (or (slot-value object slot) :null))))))

Try it out.

(serialize t (make-instance 'user :name "John"))

prints

{"" : "user", "name" : "John", "id" : null}

Looks good. How to encode the data type information is of course your choice.

Decoding works differently. A JSON object is parsed and converted into a Lisp data structure as per the *object-as* special variable. Then you provide a *decode-object-hook* function to convert this Lisp data structure into your user defined data type.

We define two convenience functions.

(defun oref (alist key)
  (cdr (assoc key alist :test #'string=)))

(defun tr (value)
  (case value
    (:true t)
    (:false nil)
    (:null nil)
    (t value)))

Now we define the *decode-object-hook* function.

(defun object-decoder (alist)
  (let ((class (oref alist "")))
    (cond ((equal class "user")
           (make-instance 'user
                          :name (oref alist "name")
                          :id (tr (oref alist "id"))))
          ;; No match, return argument as is.
          (alist))))

Try it out.

(let* ((*decode-object-hook* #'object-decoder)
       (inp (make-instance 'user :name "John"))
       (outp (parse (serialize nil inp))))
  (list (slot-value outp 'name)
        (slot-value outp 'id)))

 ⇒ ("John" nil)

That’s it. Any questions?

Performance

Don’t trust any benchmark you haven’t forged yourself!

File citm_catalog.json has a size of 1.6 MiB and contains a nice mix of objects, arrays, strings, and numbers. All libraries read the file contents from a string. For writing, there are two cases. Those libraries who can write to a stream write to the null device. The other libraries (Jonathan and json) return a string and have to carry the additional memory payload.

Note: Jzon fails to load with Clozure CL.

File large.json tests the stream I/O capabilities of the libraries. It is slightly larger than 100 MiB and is read from a file and written to the null device.

Notes:

• Jonathan and jsown can neither read from a stream nor write to a stream.
• CL-JSON fails reading on Clozure CL with the error message “No character corresponds to code #xD83D”.
• json-streams fails reading with the error message “Number with integer syntax too large 505874924095815700”.
• Jzon fails to load with Clozure CL.
• ST-JSON fails writing on Clozure CL with the error message “The value NIL is not of the expected type REAL”.
• YASON is not RFC 8259 compliant (see below).
• RS-JSON rules!

Results from the JSON Parsing Test Suite can be found here (HTML) or here (PDF). Only RS-JSON, shasht, and ST-JSON seem to be compliant (shasht and ST-JSON require some tweaking, e.g. set *read-default-float-format* to double-float). Crashes (red) and timeouts (gray) indicate serious conditions not handled by the library, for example a stack overflow or out of memory.