biqar / Privateer

Multi-versioned memory-mapped data stores for high-performance data science

Privateer

Privateer is a general-purpose data store that optimizes the tradeoff between storage space utilization and I/O performance. Privateer uses memory-mapped I/O with private mapping and an optimized writeback mechanism to maximize write parallelism and eliminate redundant writes; it also uses content-addressable storage to optimize storage space via de-duplication.

Getting Started

Privateer consists of header files that are included under privateer/include/privateer/.

Requirements

• GCC 8.3 or higher with openmp enabled.

• Boost (Tested with 1.77.0).

Building Privateer

To build an application that uses Privateer, add the Privateer headers path to the include path using "-I" compliler option or CPLUS_INCLUDE_PATH.

Building and Running Test Examples

mkdir build && cd build
cmake -DCMAKE_INSTALL_PREFIX=. ..
make install
cd build/test_apps
./<app_name>/<app_executable> <app_args>

Using Privateer

Including Privateer

#include<privateer.hpp>

Creating and memory-mapping a new data store

  Privateer privateer(addr, blocks_dir_path, version_metadata_path, max_capacity);
  void* data = privateer.data();

Opening and memory-mapping an existing data store

  Privateer privateer(addr, blocks_dir_path, version_metadata_path);
  void*	data = privateer.data();

Writeback

  privateer.msync();

Contact

• Karim Youssef (karimy at vt dot edu)
• Keita Iwabuchi (kiwabuchi at llnl dot gov)
• Roger A Pearce (rpearce at llnl dot gov)

License

Privateer is distributed under the terms of the MIT license.

SPDX-License-Identifier: MIT

Release

LLNL-CODE-827155