Snailpacks

Snailpacks is an independently maintained repository of Spack recipes oriented towards development of multimedia & gaming applications.

It also attempts to provide packages for the Scopes programming language; both the compiler and language specific packages (extensions in Spack parlance).

Getting Started

If you want to just run this to install spack locally with all the necessary configuration:

curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/salotz/snailpacks/master/bootstrap.sh | sh

Installing Spack

To leverage snailpacks, you will need to install Spack itself: instructions.

TL;DR:

git clone -c feature.manyFiles=true https://github.com/spack/spack.git ~/spack

Then in your .profile add:

. ~/spack/share/spack/setup-env.sh

You should then be able to install things:

spack install unzip
spack load unzip
which unzip

You should see that you are using the unzip installed by spack:

~/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.3.0/unzip-6.0-6gr5zye3yfrlcl23igrdasovl6wsdpyw/bin/unzip

This is a recipe that comes bundled with Spack itself.

So if you try to see info about the Scopes recipe it will be missing:

spack info scopes

Build Cache

You will have noticed that Spack compiles not just the package you asked for, but also all of its dependencies that don’t exactly match what has been solved for.

To avoid having to do this compilation yourself you can use a buildcache. This is a popular public one: https://oaciss.uoregon.edu/e4s/inventory.html

To add it to your configuration run:

spack mirror add E4S https://cache.e4s.io
spack buildcache keys -it

This should reduce the need to compile some things, but it won’t completely eliminate it.

Externals

If you would like you can have Spack use so-called “external” packages. These are basically just things you already have on your system and will prevent Spack from recompiling them if the exact version found is the same.

spack external find

This is usually fine for build dependencies, but runtime you should just stick with the Spack built ones.

There is one important external, LLVM, which will be discussed elsewhere as it is quite painful to build.

Installing Snailpacks

Snailpacks leverages the thousands of packages in the builtin repository, but adds and patches new recipes.

Installing snailpacks means adding it as a ‘repo’ to your spack installation:

mkdir -p ~/.spack/repos
git clone git@github.com:salotz/snailpacks.git ~/.spack/repos/snailpacks
spack repo add ~/.spack/repos/snailpacks

You should be able to see info on Scopes now:

spack info scopes

Scopes Programming

Compiling & Installing Scopes

Installing and using Scopes with Snailpacks is one of the primary use cases so we will discuss it here.

You should be able to just install Scopes with:

spack install scopes

Just be aware that this will compile LLVM (and all of its dependencies) from scratch. This will take a while (a few hours on a modern laptop), and it will use all of the cores on your machine.

If you want to avoid compiling LLVM see the section: Using a prebuilt LLVM compiler

However, once you do this you will not need to recompile LLVM (or any other dependencies) unless you bump the version of LLVM. Currently the version is fixed to a particular LLVM version, however this will likely change as Scopes matures.

The other thing to be aware of is that currently (as of 2022-03-23) the Scopes recipe is always built from the tip of the repository.

To get the latest you will need to remove the source cache and reinstall:

rm -rf ~/resources/spack/var/spack/cache/_source-cache/hg/~duangle/scopes
spack uninstall scopes
spack install scopes

In the near future there should be some stable versions that get tagged and only testers and those wanting the bleeding edge will need to build from tip:

spack install scopes@tip

Using a prebuilt LLVM compiler

Compiling LLVM can take a long time and may flat-out fail on machines with a small amount of RAM. To avoid this, if you can find a prebuilt version of LLVM, download it, and register it as an “external” in Spack you just have that used instead of building from scratch.

This example is for the Ubuntu Linux build.

First download and unpack the binary and packages to a prefix folder:

clang_version='12.0.1'
clang_filename="clang+llvm-${clang_version}-x86_64-linux-gnu-ubuntu-16.04"
clang_url="https://github.com/llvm/llvm-project/releases/download/llvmorg-${clang_version}/${clang_filename}.tar.xz"
sudo mkdir -p /opt/llvm
cd /opt/llvm
sudo wget "${clang_url}"
sudo tar -xvf ${clang_filename}.tar.xz
sudo rm ${clang_filename}.tar.xz

Then add this to the packages.yaml (most likely $HOME/.spack/packages.yaml):

packages:
  llvm:
    externals:
    - spec: llvm@12.0.1
      prefix: /opt/llvm/clang+llvm-12.0.1-x86_64-linux-gnu-ubuntu-

You should be able to install LLVM or Scopes and get a message like:

[+] /opt/llvm/clang+llvm-12.0.1-x86_64-linux-gnu-ubuntu- (external llvm-12.0.1-zqkt7ur7hnnbquw3w7oobhcskoxvada6)

Indicating its using the external one.

Setting Up a Project

Scopes allows for importing of modules in Scopes itself, as well as loading headers and library objects of compiled C code.

To manage this for a project we recommend setting up a virtual environment with Spack and then writing your __env.sc file to always use the spack env prefix onto the paths.

First write a spack.yaml file with the dependencies that you want (you can also use the `spack add` API):

spack:
  specs:
  - scopes
  view: true
  concretization: together

Then run these commands:

spack env create -d .
spacktivate .
spack install

You will see a folder: .spack-env/view which is a “view” of the environment which contains a unixy prefix with all the dependencies symlinked in.

Then you can simply have a __env.sc file in the same directory:

let spack-path = "/.spack-env/view"

'bind-symbols __env

    module-search-path =
        cons
            .. module-dir spack-path "/lib/scopes/packages/?.sc"
            .. module-dir spack-path "/lib/scopes/packages/?/init.sc"
            __env.module-search-path

    include-search-path =
        cons
            .. module-dir spack-path "/include"
            __env.include-search-path

    library-search-path =
        cons
            .. module-dir spack-path "/lib"
            __env.library-search-path

Now anything run with this project file, will automatically find all of libraries and headers.

See the examples/scopes folder.

C/C++ Programming

You can also use Spack to install dependencies for your C/C++ projects as well.

In the examples folder there are a couple of examples of how to configure different build systems to find the Spack installed files. Including:

[X] meson: c-meson
[X] premake: c-premake
[X] genie: c-genie
[X] cmake: c-cmake
[X] SCons: c-scons

Developing Recipes

To develop this install a separate Spack instance:

mkdir -p ./.spack-install
git clone -b v0.17.1 -c feature.manyFiles=true https://github.com/spack/spack.git ./.spack-install/spack
spack="./.spack-install/spack/bin/spack"
${spack} repo add "."

When you are done you can remove it:

${spack} repo rm snailpacks

salotz / snailpacks