pmemfile

Pmemfile project's goal is to provide low-overhead userspace implementation of file APIs using persistent memory. It consists of 2 major compoments:

libpmemfile-posix - provides syscall-like API which can be used by applications
libpmemfile - provides transparent access to libpmemfile-posix pools

How to build

Requirements:

cmake >= 3.3
libpmemobj-dev(el) >= 1.3 (http://pmem.io/nvml/)
libsyscall_intercept-dev(el) (https://github.com/pmem/syscall_intercept)

$ mkdir build
$ cd build
$ cmake .. -DCMAKE_INSTALL_PREFIX=/usr
$ make
$ make install

When developing:

$ ...
$ cmake .. -DCMAKE_BUILD_TYPE=Debug -DDEVELOPER_MODE=1 -DTEST_DIR=/mnt/pmem/pmemfile-tests
$ ...
$ ctest --output-on-failure

Note that in Debug mode "make install" installs only debug libraries.

Pmemfile-specific cmake variables:

BUILD_LIBPMEMFILE=0 - disables building of libpmemfile.so
BUILD_LIBPMEMFILE_POP=1 - builds tests using with libpmemfile-posix-over-POSIX library which imitates libpmemfile-posix.so, but uses POSIX functions provided by OS
DEVELOPER_MODE=1 - enables coding style, whitespace, license checks and enables fail-on-warning flags
LONG_TESTS=1 - enables tests which take much more time
TEST_DIR=/mnt/pmem/test_dir - provides directory where tests will create its pools
TRACE_TESTS=1 - dumps more info when test fails (requires cmake >= 3.4)
TESTS_USE_FORCED_PMEM=1 - allows tests to force enable or force disable use of optimized flush in libpmemobj (to speed them up)
USE_ASAN=1 - enables AddressSanitizer (only for debugging)
USE_UBSAN=1 - enables UndefinedBehaviorSanitizer (only for debugging)
LIBPMEMFILE_VALIDATE_POINTERS=1 - build-in support for PMEMFILE_PRELOAD_VALIDATE_POINTERS environment variable

Package for Debian-based distros

$ ./utils/build-deb.sh

Resulting packages will be in ./build-deb/release and ./build-deb/debug

Package for RPM-based distros

$ ./utils/build-rpm.sh

Resulting packages will be in ~/rpmbuild/RPMS.

Note: Before 1.0 pmemfile-posix API and ABI is considered unstable. We reserve the right to change the on-media layout without warning.

Description:

Example:

$ mkfs.pmemfile /dev/dax0.0 0
$ mkdir /tmp/mountpoint

$ sudo pmemfile-mount /dev/dax0.0 /tmp/mountpoint
$ alias pf='LD_PRELOAD=libpmemfile.so'

# or if you don't have root access:

$ alias pf='LD_PRELOAD=libpmemfile.so PMEMFILE_POOLS=/tmp/mountpoint:/dev/dax0.0'

# now all commands prefixed with 'pf' will see files under /tmp/mountpoint,
# but files will be stored in file system backed by /dev/dax0.0
$ pf mkdir /tmp/mountpoint/dir_in_pmemfile
$ pf cp README.md /tmp/mountpoint/dir_in_pmemfile
$ pf ls -l /tmp/mountpoint/
total 0
drwxrwxrwx 2 user group 4008 Feb 16 17:46 dir_in_pmemfile
$ pf ls -l /tmp/mountpoint/dir_in_pmemfile
total 16
-rw-r--r-- 1 user group 1014 Feb 16 17:46 README.md
$ pf cat /tmp/mountpoint/dir_in_pmemfile/README.md | wc -c
1014
# verify that files are stored in pmemfile pool:
$ ls -l /tmp/mountpoint/
total 0
$ ls -l /tmp/mountpoint/dir_in_pmemfile
ls: cannot access '/tmp/mountpoint/dir_in_pmemfile': No such file or directory

Limitations:

libpmemfile[-posix].so does not support multiple processes accessing the pool (libpmemobj limitation)
libpmemfile.so supports Linux only (other UNIX-like systems could be supported)
libpmemfile.so supports x86_64 only (libsyscall_intercept, libpmem and libpmemobj limitation)
libpmemfile.so is not safe with remotely replicated pool (libpmemfile-posix.so has no such limitation)

Debugging:

Environment variables:

libsyscall_intercept.so:

INTERCEPT_LOG - log file for all intercepted syscalls (default: none)
INTERCEPT_LOG_TRUNC - when set to 0, the above log file is not truncated (default: 1)
INTERCEPT_HOOK_CMDLINE_FILTER - intercept syscalls only when process name matches this variable (used to skip gdb from intercepting) (default: empty)

libpmemobj.so:

PMEMOBJ_LOG_FILE - log file (default: none)
PMEMOBJ_LOG_LEVEL - logging level (default: 0)

libpmemfile-posix.so:

PMEMFILE_POSIX_LOG_FILE - log file (default: none)
PMEMFILE_POSIX_LOG_LEVEL - logging level (default: 0)

libpmemfile.so:

PMEMFILE_PRELOAD_LOG - log file (default: none)
PMEMFILE_PRELOAD_LOG_TRUNC - when set to 0, the above log file is not truncated (default: 1)
PMEMFILE_EXIT_ON_NOT_SUPPORTED - when set to 1, aborts an application which uses unsupported syscall (default: 0)
PMEMFILE_PRELOAD_PAUSE_AT_START - pauses initialization of pmemfile until debugger is attached (default: 0)

Other variables:

PMEMFILE_BLOCK_SIZE - forces one block size (default: dynamic)
PMEMFILE_CD - performs early chdir() to specified directory, used as a workaround for missing multi-process support when application must start from pmemfile-backed directory (default: none)
PMEMFILE_IGNORE_INODE_FREE_ERRORS - when set to 1, disables abort() when freeing inode's metadata fails (it defers freeing to the next application start) - can be used to get out of out-of-space situations (default: 0)
PMEMFILE_OVERALLOCATE_ON_APPEND - when set to 0, disables allocation of more space than required (default: 1)
PMEMFILE_PRELOAD_PROCESS_SWITCHING - when set to 1, enables VERY slow emulation of multi-process support, used for testing pmemfile with file system test suites (default: 0)
PMEMFILE_PRELOAD_VALIDATE_POINTERS - when set to 1, verifies memory reaching libpmemfile through syscall arguments is accessible; it's very slow, so it should never be used in production for non-buggy applications

Other stuff

vltrace - tool for tracing applications and evaluating whether libpmemfile.so supports them (https://github.com/pmem/vltrace)

LitongYou / pmemfile