Simplified Interface to Complex Memory
This project is split into two interfaces: low
and high
.
The low
interface provides a minimal interface for application wanting to
manage their own memory on heterogeneous memory tiers. It also provides an
arena allocator that application developers can use to create jemalloc
arenas
on different memory tiers and allocate to those tiers.
The high
interface attempts to automatically manage the memory tiers for the
application. It provides an LLVM compiler pass (and compiler wrappers) to
automatically transform applications to make the appropriate high
interface
calls, as well as a runtime library which provides profiling for the
application. The profiling is currently meant to be used offline; that is,
after enabling the profiling for an application run, the results are printed
out at the end of the run, and that information must be fed into a second run
to make use of it. An online approach is planned.
The only dependency that you will need for the low-level interface is
jemalloc
. The build system will automatically look in /usr
for this
dependency, but if you want to provide a custom location, you can pass the
--with-jemalloc
argument to configure
.
For the high-level interface, you need an installation of LLVM. LLVM 4.0 and
later have been tested, although 3.9 may possibly work. For the profiling, you
will also need an installation of libpfm
, which is a small helper library for
perf
that is available on most distributions. These two dependencies are
found automatically, but again, the user can specify --with-llvm=
and
--with-libpfm
to use a custom location.
jemalloc
and llvm
can be installed by running install_deps.sh
with
the appropriate command line options. In order to assist with the automation
of building and running SNAP, OpenMPI-3.1.1
can also be installed through
this script.
As an example for using your package manager to install the dependencies, Ubuntu Trusty requires the following packages. You can replace the GCC version with whichever you want, or use Clang instead.
g++-8
gfortran-8
libhwloc-dev
libiomp-dev
libnuma-dev
libpfm4
libpfm4-dev
llvm-3.9-dev
numactl
If you have a version of LLVM newer than 3.9, you can likely omit libiomp-dev
in exchange for the builtin OpenMP implementation.
On Debian Stable (9.5.0 at time of writing), the required packages (for both low- and high-level interfaces) are:
autoconf
automake
g++
gcc
gfortran
git
libnuma-dev
libpfm4-dev
libtool
libtool-bin
llvm-4.0
llvm-4.0-dev
make
In general, to compile and install on a new system, run
./autogen.sh
./configure --prefix=[DIR]
make
make install
On a fresh Debian Stable (9.5.0) system, SICM can easily be installed with these commands:
sudo apt-get install git autoconf automake make gcc libpfm4-dev llvm-4.0-dev llvm-4.0 libtool g++ libnuma-dev libtool-bin gfortran
git clone https://github.com/lanl/SICM.git
cd SICM
./install_deps.sh --no-llvm
./autogen.sh
./configure --prefix=$(readlink -f ./deps) --with-jemalloc=$(readlink -f ./deps) --with-llvm=$(llvm-config-4.0 --prefix)
make -j$(nproc --all)
make install
Function Name | Description |
---|---|
sicm_init |
Detects all memory devices on system, returns a list of them. |
sicm_fini |
Frees up a device list and associated SICM data structures. |
sicm_find_device |
Return the first device that matches a given type and page size. |
sicm_device_alloc |
Allocates to a given device. |
sicm_device_free |
Frees memory on a device. |
sicm_can_place_exact |
Returns whether or not a device supports exact placement. |
sicm_device_alloc_exact |
Allocate memory on a device with an exact base address. |
sicm_numa_id |
Returns the NUMA ID that a device is on. |
sicm_device_page_size |
Returns the page size of a given device. |
sicm_device_eq |
Returns if two devices are equal or not. |
sicm_move |
Moves memory from one device to another. |
sicm_pin |
Pin the current process to a device's memory. |
sicm_capacity |
Returns the capacity of a given device. |
sicm_avail |
Returns the amount of memory available on a given device. |
sicm_model_distance |
Returns the distance of a given memory device. |
sicm_is_near |
Returns whether or not a given memory device is nearby the current NUMA node. |
sicm_latency |
Measures the latency of a memory device. |
sicm_bandwidth_linear2 |
Measures a memory device's linear access bandwidth. |
sicm_bandwidth_random2 |
Measures random access bandwidth of a memory device. |
sicm_bandwidth_linear3 |
Measures the linear bandwidth of a memory device. |
sicm_bandwidth_random3 |
Measures the random access bandwidth of a memory device. |
Function Name | Description |
---|---|
sicm_arenas_list |
List all arenas created in the arena allocator. |
sicm_arena_create |
Create a new arena on the given device. |
sicm_arena_destroy |
Frees up an arena, deleting all associated data structures. |
sicm_arena_set_default |
Sets an arena as the default for the current thread. |
sicm_arena_get_default |
Gets the default arena for the current thread. |
sicm_arena_get_device |
Gets the device for a given arena. |
sicm_arena_set_device |
Sets the memory device for a given arena. Moves all allocated memory already allocated to the arena. |
sicm_arena_size |
Gets the size of memory allocated to the given arena. |
sicm_arena_alloc |
Allocate to a given arena. |
sicm_arena_alloc_aligned |
Allocate aligned memory to a given arena. |
sicm_arena_realloc |
Resize allocated memory to a given arena. |
sicm_arena_lookup |
Returns which arena a given pointer belongs to. |
The high-level interface is normally used with the compiler wrappers located in
bin/
. Users should use these wrappers to compile their applications, and a
compiler pass will automatically transform the code so that it calls the
high-level interface with the appropriate arguments, including initialization,
destruction, and the proper allocation functions. Assuming the high-level
interface is linked to the application as a shared library, it automatically
initializes itself. All heap allocation routines are replaced by calls to
void* sh_alloc(int id, size_t sz)
, which associates an ID with a given
allocation and allocates the memory into an arena with other allocations of
that ID.
- All blocks use curly braces
- Even one-line blocks
- Constants on the left side of
==
if(NULL == foo) { ...
- Functions with no arguments are
(void)
- No C++-style comments in C code
- No GCC extensions except in GCC-only code
- No C++ code in libraries
- Discouraged in components
- Always define preprocessor macros
- Define logicals to 0 or 1 (vs. define or not define)
- Use
#if FOO
, not#ifdef FOO