pstore is a lightweight persistent append-only key/value store intended for use as a back-end for the LLVM Program Repository.

Its design goals are:

Performance approaching that of an in-memory hash table
Good support for parallel compilations
Multiple indices
In-process

Building pstore
- Prerequisites
- Building
  - Standalone
  - Inside LLVM
Getting started
- Using the read and write utilities

Building pstore

Prerequisites

pstore is built and tested on a variety of platforms:

Ubuntu Linux 20.04.3 LTS: building with GCC 7.5.0, GCC 10.3.0, Clang 8.0.1, and Clang 12.0.0. We also build with the official GCC 5.5.0 docker image.
macOS: building with Xcode 13.2
Windows: building with Visual Studio 2019 and 2022

In addition, there’s support for FreeBSD 11, NetBSD 9.1, and Solaris 11.4.

To build it, you’ll also need the following tools:

cmake (version 3.4 or later, version 3.8 or later if using Visual Studio)

Optionally:

Doxygen for building the documentation
GraphViz for graph rendering
Node.js for the ElectronJS-based broker dashboard and some system tests.
Python 2.7 or later for running the system tests as well as a few utilities
Valgrind for extra validation of the executables (enabled by passing the -D PSTORE_VALGRIND=Yes argument when running cmake to generate the build)

Building

pstore may be built either as a standalone collection of libraries, header files, and utilities or as a project within the LLVM framework.

Standalone

The pstore build system uses cmake. If you’re not very familiar with cmake, there’s a small utility (found in utils/make_build.py) which will create an out-of-tree directory in which to build the project and run cmake with the correct arguments for your platform.

python ./utils/make_build.py
cmake ‑‑build build_linux

The build directory will be one of build_linux, build_mac, build_win32, and so on.

Inside LLVM

Make sure that pstore is located within the llvm-project directory tree. For example, to build pstore inside LLVM with Program Repository Support (llvm-project-prepo):

git clone http://github.com/SNSystems/llvm-project-prepo.git
cd llvm
git clone http://github.com/SNSystems/pstore.git
cd -

Build LLVM as normal enabling the pstore subproject in addition to any others. For example:

mkdir build
cd build
cmake -G Ninja \
      -D LLVM_ENABLE_PROJECTS="clang;pstore" \
      -D LLVM_TARGETS_TO_BUILD=X86 \
      -D LLVM_TOOL_CLANG_TOOLS_EXTRA_BUILD=Off \
      ../llvm
ninja

Getting started

Using the read and write utilities

The pstore-read and pstore-write tools provide a simple way to experiment with the capabilities of the pstore library. Consider the following exchange:

$ echo foo > foo.txt
$ echo bar > bar.txt
$ pstore-write --add-file mykey,foo.txt pstore.db
$ pstore-read pstore.db mykey
foo
$ pstore-write --add-file mykey,bar.txt pstore.db
$ pstore-read pstore.db mykey
bar
$ pstore-read --revision=1 pstore.db mykey
foo
$ pstore-dump --log pstore.db
---

- file : 
      path : pstore.db
      size : 488
  log  : 
      - { number: 2, size: 56, time: 2020-11-19T15:24:04Z }
      - { number: 1, size: 56, time: 2020-11-19T15:23:52Z }
      - { number: 0, size: 0, time: 2020-11-19T15:23:52Z }
...
$

Let’s pick this apart one step at a time…

$ echo foo > foo.txt
$ echo bar > bar.txt

Create two files which contain short text strings that we’ll shortly record in a pstore file:

$ pstore-write --add-file mykey,foo.txt pstore.db

This command creates a new pstore file named pstore.db or appends to that file if it already exists. The choice of file name is arbitrary. The tool creates an entry in one of the pstore indexes with the key “mykey” which has a corresponding value string “foo\n” as read from the foo.txt file we created earlier.

$ pstore-read pstore.db mykey
foo

Next we use the pstore-read utility to search the pstore.db file for key named “mykey” and print its value. Happily it prints “foo\n”: the same string that we passed to pstore-write above.

$ pstore-write --add-file mykey,bar.txt pstore.db

Now we run pstore-write again but this time associating the value “bar” (plus the inevitable newline) with the key “mykey”.

$ pstore-read pstore.db mykey
bar

Running pstore-read a second time prints “bar” showing that the key “mykey” has been updated.

$ pstore-read --revision=1 pstore.db mykey
foo

This command is a little more interesting. Here we’ve retrieved the original value that was linked to “mykey”. Each time that pstore-write stores data in a pstore file, it does so in a self-contained transaction. Each transaction is appended to the file, preserving the previous contents. The first transaction in a pstore file is number 0 (which is always empty). The first time data is added, transaction 1 is created; the second time, we build transaction 2, and so on. Any redundant data stays in the file — and is immutable — until the garbage collector (pstore-vacuumd) runs. This property enables a store to be read without any need for locks.

$ pstore-dump --log pstore.db

The pstore-dump command allows us to inspect the innards of a pstore file. It produces a YAML dump of the requested structures: the transaction log in this case showing a list of all of the transactions in the file (newest first), how much data each of them is carrying, and when that data was committed. There is a small number of other utilities in the tools/ directory which allow various aspects of a pstore file to be explored.

SNSystems / pstore

Table of Contents