Faasm Python Environment

This build cross-compiles CPython and a number of Python modules to WebAssembly for use in Faasm.

It also provides pyfaasm, a small Python library which uses ctypes to support calls to the Faasm host interface.

Development

Most use of this project is via the Faasm development environment.

You should only need the instructions below if you want to:

Modify the Faasm CPython runner
Change the Faasm Python host interface (pyfaasm).
Add Python modules to the Faasm environment.

Building CPython and libraries

Faasm runs python code by cross-compiling the CPython runtime to WebAssembly, and adding a small entrypoint function to run Python code on the cross-compiled runtime.

To cross-compile the CPython runtime, we first need to install a native CPython of the exact same version:

inv cpython.native

Then, you can cross-compile CPython from our fork:

inv cpython.wasm

This generates a static version of libpython that we link to the entrypoint function. To cross-compile this entrypoint function you can run:

inv cpython.func

Change the Python host interface

Faasm provides a small python library, pyfaasm so that functions written in Python (which are not cross-compiled to WebAssembly) can communicate with the Faasm runtime.

To install pyfaasm we need to use the same pip version we installed natively (and cross-compiled) as part of the CPython build in the previous section. Setuptools and distutils, pip's tooling to install libraries, interrogate the system environment during the library install process. This makes it quite difficult to install pyfaasm at the right location, using the right version of pip. We use crossenv to help with that.

To install pyfaasm we need to activate the crossenv virtual environment, we do so in a separate shell inside the container for simplicity:

bash
./bin/crossenv_setup.sh
source ./cross_venv/bin/activate
pip3 install -r crossenv/requirements.txt
inv -r crossenv modules.build
exit

To use the pyfaasm library in Faasm, we still need to copy the installed files to the right runtime location:

inv modules.install

Adding Python modules to the Faasm environment

Crossenv picks up the cross-compilation environment from the CPython build artifacts. With the crossenv activated, we can build modules with normal pip. However, there is a wrapper script that will apply modifications if we know about them:

bash
./bin/crossenv_setup.sh
source ./cross_venv/bin/activate

# Build all supported modules
inv -r crossenv modules.build

# Install experimental modules
inv -r crossenv modules.build --experimental

# Install numpy
inv -r crossenv modules.build --name numpy

# (Attempt) to install arbitrary module
inv -r crossenv modules.build --name <module_name>
exit

Libraries will then be installed to cross_venv/cross/lib/python3.8/site-packages. To install them in the Faasm sysroot, you can then run:

inv modules.install

Debugging module builds

You can debug module builds by running python setup.py install through your debugger.

You can also set DISTUTILS_DEBUG=1 to get distutils to print more info.

Experimental modules

Some of the modules are experimental, these may require some extra set-up.

MXNet and Horovod

To install the Python MXNet module we first need to cross-compile the MXNet shared library:

# Update all submodules
cd third-party/mxnet
git submodule update --init
cd ../horovod
git submodule update --init

# Run our MXNet cross-compile (outside crossenv)
cd ../..
inv mxnet

Then we can install mxnet and horovod:

. ./cross_venv/bin/activate
inv libs.install --name mxnet
inv libs.install --name horovod

To clean and uninstall:

# Clean then rebuild
inv mxnet --clean

# Uninstall mxnet
inv mxnet.uninstall

Numpy

Faasm's NumPy build relies on BLAS and LAPACK support. The right cross-compiled libraries should be picked up by numpy due to the addition of the site.cfg .

22/12/2022 - NumPy support is currently broken

Code changes

The CPython build uses this slightly modified fork of CPython.

To see the changes made to CPython, see this compare.

A similar (small) list of changes for numpy can be seen here.

CPython is built statically, some notes on this process can be found here.

Several of the code changes to CPython and numpy were borrowed from pyodide.

Releasing

This repo gets built as a container, faasm/cpython. If you want to release a new version, you can:

Bump the code version with: inv git.bump
Commit to your branch
Run inv git.tag
Check the release build has run
Create a pull request

The release build will generate a docker image with the new tag. You can also trigger the image build manually with:

inv docker.build [--push] [--nocache]

faasm / python