Quickstart with litgen

litgen_template is a template repository to build python bindings using litgen, and scikit-build.

This template is based on scikit_build_example.

Usage for final users

Below are the instructions you would give to final users of your bindings. They are extremely short:

First, install the package from source

git clone https://github.com/pthom/litgen_template.git && cd litgen_template
pip install -v .

Then, use it from python

import daft_lib
daft_lib.add(1, 2)

(this template builds bindings for a C++ library called DaftLib, and publishes it as a python module called daft_lib)

Of course, you could also publish your bindings to PyPI, and tell your users to install them with pip install daft-lib. This template provides tooling to make the publishing process easier, via cibuildwheel

Autogenerate the binding code

Install requirements

Create a virtual environment

python3 -m venv venv
source venv/bin/activate

Install the requirements

pip install -r requirements-dev.txt

This will install litgen (the bindings generator), pybind11 (a library to create C++ to Python bindings), pytest (for the tests), black (a code formatter), and mypy (static type checker for python).

See requirements-dev.txt.

Generate bindings

Optionally, change the C++ code

Change the C++ code (add functions, etc.) in src/cpp_libraries/DaftLib
Adapt the generation options inside tools/autogenerate_bindings.py

Run the code generation via litgen

python tools/autogenerate_bindings.py

This will:

Write the cpp binding code into src/python_bindings/pybind_DaftLib.cpp
Write the python stubs (i.e. typed declarations) inside src/python_bindings/daft_lib/__init__.pyi.

Tip: compare the python stubs with the C++ header file to see how close they are!

Note: the options inside autogenerate_bindings.py showcase a subset of litgen customization capabilities. See the litgen documentation for more details. They are heavily documented, and correspond to the documentation you can find in DaftLib.h

Adapt for your own library

Names, names, names

In this template repository:

the C++ library is called DaftLib
the native python module generated by pybind11 is called _daft_lib
the python module which is imported by users is called daft_lib (it imports and optionally adapts _daft_lib)
the pip package that can optionally be published to PyPI is called daft-lib (as Pypi does not allow dashes in package names)

You can change these names by running change_lib_name.py in the tools/change_lib_name folder.

Structure of this template

Bound C++ library

The C++ library DaftLib is stored inside src/cpp_libraries/DaftLib/

src/cpp_libraries/
└── DaftLib/
    ├── CMakeLists.txt
    ├── DaftLib.h
    └── cpp/
        └── DaftLib.cpp

Python bindings

The python bindings are stored inside src/python_bindings/

src/python_bindings/
      ├─── module.cpp              # Main entry point of the python module
      │── pybind_DaftLib.cpp       # File with bindings *generated by litgen*
      │
      └─ daft_lib/
          ├── __init__.pyi         # Stubs *generated by litgen*
          ├── __init__.py          # The python module (daft_lib) main entry point
          │                        # (it imports and optionally adapts _daft_lib)
          ├── py.typed             # An empty file that indicates that the python module is typed
          │
          └── _daft_lib.xxx.so*    # (optional: the native _daftlib module generated by pybind11,
                                   #  only present in editable mode)

Tooling for the bindings generation

tools/
├── autogenerate_bindings.py
└── change_lib_name/
    └── change_lib_name.py

tools/autogenerate_bindings.py is the script that will generate the bindings using litgen.

tools/change_lib_name/change_lib_name.py is an optional utility that you can use once after cloning this template, in order to rename the libraries (e.g. from DaftLib to MyLib, daft_lib to my_lib, etc.)

Compilation

├── CMakeLists.txt                 # CMakeLists (used also by pip, via skbuild)
├── litgen_cmake/                  # litgen_setup_module() is a cmake function that simplifies
│   └── litgen_setup_module.cmake  # the deployment of a python module with litgen
│
├── requirements-dev.txt           # Requirements for development (litgen, pybind11, pytest, black, mypy)

Deployment

pyproject.toml is used by pip and skbuild to build and deploy the package. It defines the name of the package, the version, the dependencies, etc.

Continuous integration

Several github workflows are defined in .github/workflows:

.github/
├── dependabot.yml   # Configuration for dependabot (automatically update CI dependencies)
├── workflows/
    ├── conda.yml    # Build the package with conda
    ├── pip.yml      # Build the package with pip
    └── wheels.yml   # Build the wheels with cibuildwheel, and publish them to PyPI
                     # (when a new release is created on github)

Note:

cibuildwheel is configurable via options defined in the pyproject.toml file: see the [tool.cibuildwheel] section.
it is also configurable via environment variables, see cibuildwheel documentation

Tests

├── tests/daft_lib_test.py    # This is a list of python tests that will check
└── pytest.ini

Those tests are run by cibuildwheel and by the pip CI workflow.

Editable development mode

If you want to quickly iterate on the C++ code, and see the changes reflected in python without having to reinstall the package, you should use the python editable development mode.

Setup editable mode

Step1: Install the package in editable mode

pip install -v -e .  # -e stands for --editable, and -v stands for --verbose

Step 2: Create a standard C++ build directory

mkdir build && cd build
cmake ..
make # rebuild when you change the C++ code, and the changes will be reflected in python!

Debug C++ bindings in editable mode

The pybind_native_debug executable provided in this template is a simple C++ program that can be used to debug the bindings in editable mode.

src/pybind_native_debug/
        ├── CMakeLists.txt
        ├── pybind_native_debug.cpp
        └── pybind_native_debug.py

Simply edit the python file pybind_native_debug.py by adding calls to the C++ functions you want to debug. Then, place breakpoints in the C++ code, and debug the C++ program.

Development tooling

This template is ready to be used with additional tools:

pre-commit
ruff
mypy
pyright
black
pytest
cibuildwheel

pre-commit

pre-commit is a tool that allows you to run checks on your code before committing it. This template provides a default pre-commit configuration for it, but it is not active by default;

You can install pre-commit with:

pip install pre-commit

Then, you can activate the pre-commit hooks for your repository with:

pre-commit install

The pre-commit configuration file .pre-commit-config.yaml, is configured with the following hooks:

basic sanity checks: trailing-whitespace, end-of-file-fixer,check-yaml, check-added-large-files
black: uncompromising Python code formatter
ruff: fast Python linter and code formatter (only used for linting)
mypy: static type checker for python

You can find more interesting hooks on the pre-commit hooks repository, and for example add ruff, mypy, black, etc.

You may want to disable some checks in the .pre-commit-config.yaml file if you think this is too strict for your project.

ruff: python linter and code formatter

ruff is a very fast python linter and code formatter. You can install it and run it with:

pip install ruff # install ruff (once)
ruff . # each time you want to check your python code

mypy and pyright: static type checkers for python

mypy and pyright are static type checkers for python.

You can use either one of them, or both.

mypy

pip install mypy  # install mypy (once)
mypy # each time you want to check your python code

mypy is configured via the mypy.ini file.

pyright

pip install pyright # install pyright (once)
pyright # each time you want to check your python code

pyright is configured via the pyrightconfig.json file.

black: python code formatter

black is a python code formatter.

pip install black # install black (once)
black . # each time you want to format your python code

pytest: python tests

pytest is an easy-to-use python test framework.

pip install pytest # install pytest (once)
pytest # each time you want to run your python tests

It is configured via the pytest.ini file, and tests are stored in the tests folder.

cibuildwheel: build wheels for all platforms

ci-buildwheel is a tool that allows you to build wheels for all platforms.

It is configured via the pyproject.toml file (see the [tool.cibuildwheel] section), and the github workflow file.

run_all_checks

tools/run_all_checks.sh is a script you can run before committing or pushing. It will run a collection of checks (mypy, black, ruff, pytest).

pthom / litgen_template