NFiles

NFiles is a Common Lisp library that deals with customizable path resolution, file persistence and loading.

Its main use case is help manage user-centric files like configuration files.

In some aspects, it can be seen as “Common Lisp ‘logical pathnames’ over CLOS”.

Goals

Performance

Data is not persisted to disk if it does not need to.
Files are read only once (unless modified externally).

Extensibility

Persist any data structure the way you want it.

Reliability

no corruption and no data loss should occur.

Features

Dynamic and customizable path expansion.
Extensible serialization and deserialization.

Cached reads and writes: When a file object expands to the same path as another one, a read or write on it won’t do anything in case there was no change since last write.

(Experimental!) On-the-fly PGP encryption.
Profile support.
On read errors, existing files are backed up.
On write errors, no file is written to disk, the existing file is preserved.
A remote-file can point to a URL, which is automatically downloaded if the local file is not found.

Motivation

This package was developed after dealing with a problem when delivering Common Lisp images: when an image is generated, path expansion may already be resolved and thus hard-coded within the image, which makes it unfit for delivery. Consider this:

> (defvar *foo-path* (uiop:xdg-config-home))
*FOO-PATH*
> *foo-path*
#P"/home/johndoe/.config/"

Now if I ship this image to my friend Kaboom, *foo-path* will expand to

#P"/home/johndoe/.config/"

on their machine instead of the expected

#P"/home/kaboom/.config/"

Examples

A basic session:

(defvar *config-file* (make-instance 'nfiles:config-file :base-path #p"my-app/init.lisp"))

(nfiles:expand *config-file*)
; => #P"/home/johndoe/.config/my-app/init.lisp"

(setf (nfiles:content *config-file*) "Hello file!") ; The file is written to disk.

(nfiles:content *config-file*)
; => "Hello file!"

The following convenience macro ensures the file is updated when done with the body:

(nfiles:with-file-content (content *config-file*)
  (format t "Length: ~a~%" (length content))
  (setf content (serapeum:string-replace "file" content "config")))

The with-paths helper allows for let-style bindings of the expanded paths:

(let ((file1 (make-instance 'nfiles:file))
      (file2 (make-instance 'nfiles:file :base-path #p"alt")))
  (nfiles:with-paths ((path1 file1)
                      (path2 file2))
    (list path1 path2)))

A remote-file works the same but needs some specialization:

(defmethod nfiles:fetch ((profile nfiles:profile) (file remote-counter-file) &key)
  (dex:get (nfiles:url file)))

;; Optional:
(defmethod nfiles:check ((profile nfiles:profile) (file remote-counter-file) content &key)
  (let ((path (nfiles:expand file)))
    (ironclad:byte-array-to-hex-string
     (ironclad:digest-file :sha3 path))))

(let ((file (make-instance 'nfiles:remote-file
                           ;; The URL to download from if the file is not found on disk.
                           :url (quri:uri "https://example.org")
                           ;; Without base-path, the file won't be saved to disk.
                           :base-path #p"/tmp/index.html"
                           :checksum "794df316afac91572b899b52b54f53f04ef71f275a01c44b776013573445868c95317fc4a173a973e90addec7792ff8b637bdd80b1a6c60b03814a6544652a90")))
  ;; On access, file is automatically downloaded if needed and the checksum is verified:
  (nfiles:content file)
  ;; ...
  )

See the package documentation for a usage guide and more examples.

Configuration

NFiles was designed with configurability in mind. All configuration happens through subclassing of the file and profile classes together with method specialization.

All configuration methods are specialized against profile and file so that the user can easily compose the behaviour:

Profile-customization impacts all files using that profile;
File-customization impacts the files of that specific type (or subtype) regardless of their profile.

Of course you can specialize against both!

The specialization methods are divided into the following:

resolve: This is where path resolution is done. On call site, prefer the expand convenience wrapper.
deserialize and serialize: This is how the content is transformed to the file on disk. These functions are meant to be called by the read-file and write-file methods.
read-file and write-file: This is how the file is read and written to disk. These functions are responsible for calling the deserialize and serialize methods.
fetch: This generic function is only called for remote-file objects. You must define its methods. It does not have any method by default so as to not burden NFiles with undesirable dependencies.

check :: Like fetch, this generic function is only called for remote-file objects to test the integrity of the downloaded file. You must define its methods. It does not have any method by default so as to not burden NFiles with undesirable dependencies.

Conditions and restarts

Some NFiles-specific conditions are raised in case of exceptional situations to provide for interactive and customizable behaviour:

external-modification: The file was modified externally. See the on-external-modification slot to automate what to do in this case.

Read error restarts can also customized, see the on-read-error slot to automate what to do in this case.

process-error: This may be raised for instance when gpg fails to encrypt. The use-recipient restart is provided to retry with the given recipient.

Shadowing

NFiles 1 shadows cl:delete, thus you should not :use the package (as with any other library anyways).

Platform support

It’s pure Common Lisp and all compilers plus all operating systems should be supported.

Some notes:

All compilers but SBCL depend on IOlib to preserve file attributes.
Android devices also depend on IOlib to preserve file attributes, regardless of the compiler.
File attributes might not be preserved on Windows.

Roadmap

Improve PGP support.
Support OS-level locks (à-la Emacs / LibreOffice).
Improve portability.
- In particular, preservation of file attributes may not work on Windows.
Compressing write-file and read-file (for instance with zstd / lz).
- But is it such a good idea? Users should prefer compression at the level of the file system.

Change log

1.1.4

Remove NASDF as a dependency.

1.1.3

Ensure a cl:pathname is returned from resolve.
Complete some missing documentation.

1.1.2

Add restarter functions for ask, reload, etc. Mind that cl:delete is now shadowed (this was necessary to preserve backward compatibility). Do not :use the package!
Switch from hu.dwim.defclass-star to nclasses.

1.1.1

Allow path expansion for virtual-file (as in 1.0.0).
This restores the usefulness of virtual-files, namely to handle the path-expansion business while deferring the read/write business to a third-party.

virtual-profile still nullifies path expansions (as in 1.1.0).

1.1.0

Add support for Android.
Nullify path expansion for virtual-file and virtual-profile.
Ensure that the deserialize method of virtual-file and virtual-profile return nil.
Fix basename corner case.
Add report messages to all restarts.

History

NFiles was originally developed for user file management in Nyxt, so the “N” may stand for it, or “New”, or whatever poetic meaning you may find behind it!

atlas-engineer / nfiles