Subclasses issue with different headers: how to use amalgamation

[Dan]

When litgen reads a subclass that's defined in the same file as its base class, it will correctly generate bindings for all methods defined on the base class. If you move the base class to a separate header file, though, it does not.

If the base class includes a pure-virtual method which the subclass does not override, this generates an error at compile time because the generated trampoline class is also missing the PYBIND_OVERRIDE_PURE for that method.

Unfortunately, using the code_preprocess_function to replace #includes with their contents does not appear to be a viable workaround, as it results in multiple definitions for the base class.

Reproduction

Here's a python script that writes, generates, and tries to compile a minimally-reproducing example, adapted from the Dog : Animal inheritance example in the docs.

# litgen_error.py
TRIGGER_THE_BUG = True
TRIGGER_PURE_VIRTUAL_COMPILER_ERROR = False

ANIMAL_CLASS = """
#include <string>

class Animal {
public:
    virtual std::string go(int n_times) = 0;
    virtual std::string name() { return "unknown"; }
    virtual bool is_furry()%s
};
""" % (
    "=0;" if TRIGGER_PURE_VIRTUAL_COMPILER_ERROR else '{return true;}'
)

DOG_SUBCLASS = """
class Dog : public Animal {
public:
    std::string go(int n_times) override {
        std::string result;
        for (int i=0; i<n_times; ++i)
            result += bark() + " ";
        return result;
    }
    virtual std::string bark() { return "woof!"; }
};
"""

CPP_OUTPUT_STUB = """
#include "dog.hpp"
#include <pybind11/pybind11.h>
namespace py = pybind11;

// <litgen_glue_code>
// </litgen_glue_code>
void py_init_module_test_output(py::module& m) {
// <litgen_pydef>
// </litgen_pydef>
}

PYBIND11_MODULE(test_output, m) { py_init_module_test_output(m); }
"""

PYI_OUTPUT_STUB = """
# <litgen_stub>
# </litgen_stub>
"""

HEADER_FILES = ['dog.hpp']

if TRIGGER_THE_BUG:
    HEADER_FILES.append('animal.hpp')
    with open('animal.hpp', 'w') as fd:
        fd.write(ANIMAL_CLASS)

with open('dog.hpp', 'w') as fd:
    if TRIGGER_THE_BUG:
        fd.write('#include "animal.hpp"\n')
    else:
        fd.write(ANIMAL_CLASS)
    fd.write(DOG_SUBCLASS)

with open('test_output.cpp', 'w') as fd:
    fd.write(CPP_OUTPUT_STUB)

with open('test_output.pyi', 'w') as fd:
    fd.write(PYI_OUTPUT_STUB)


from litgen import LitgenOptions, write_generated_code_for_files

conf = LitgenOptions()
conf.class_override_virtual_methods_in_python__regex = '^Animal$|^Dog$'
write_generated_code_for_files(
    conf,
    input_cpp_header_files=HEADER_FILES,
    output_cpp_pydef_file='test_output.cpp',
    output_stub_pyi_file='test_output.pyi',
)

#!/bin/bash
cd $(mktemp -d)
python litgen_error.py
EXT_SUFFIX=$(python -c 'from sysconfig import *; print(get_config_var("EXT_SUFFIX"))')
g++ test_output.cpp \
  $(pkg-config --cflags --libs pybind11) \
  $(pkg-config --cflags --libs python3) \
  -fPIC -shared -o test_output$EXT_SUFFIX

Setting TRIGGER_BUG to True in the above code puts class Animal in its own file, which results in class Dog not having bindings for ::name() or ::is_fuzzy(). Setting TRIGGER_BUG to False generates the expected litgen code.

[Pascal Thomet]

Hello,

Thank you for taking the time to write a repro.

In your case, you can use an amalgamation utility I wrote. It is designed to not include twice the same file, so that it should do what you expect.

Here is a commented example:

from codemanip.amalgamated_header import AmalgamationOptions, write_amalgamate_header_file

amalgamation_options  = AmalgamationOptions()
# Base dir for the include paths
amalgamation_options.base_dir = '.'
# All possible subdirs for includes
amalgamation_options.include_subdirs = ["./"]
# Start of the local includes (so that we do not include <string> for example)
# In your example, I had to add "./" to the include path, so that it is handled by local_includes_startwith
amalgamation_options.local_includes_startwith = './'
# You wlll need a main header that includes all the API you want to expose
amalgamation_options.main_header_file = 'dog.hpp'
# Destination file for the amalgamated header
amalgamation_options.dst_amalgamated_header_file = 'amalgamated_header.hpp'
# Let's write the amalgamated header
write_amalgamate_header_file(amalgamation_options)

Beware, I had to change one line in your example:

with open('dog.hpp', 'w') as fd:
    if TRIGGER_THE_BUG:
        fd.write('#include "./animal.hpp"\n')  # I had to add "./" here, so that it is handled by local_includes_startwith
    else:
        fd.write(ANIMAL_CLASS)
    fd.write(DOG_SUBCLASS)

And if you want the full working code:

from litgen import LitgenOptions, write_generated_code_for_file

# litgen_error.py
TRIGGER_THE_BUG = True
TRIGGER_PURE_VIRTUAL_COMPILER_ERROR = False

ANIMAL_CLASS = """
#include <string>

class Animal {
public:
    virtual std::string go(int n_times) = 0;
    virtual std::string name() { return "unknown"; }
    virtual bool is_furry()%s
};
""" % (
    "=0;" if TRIGGER_PURE_VIRTUAL_COMPILER_ERROR else '{return true;}'
)

DOG_SUBCLASS = """
class Dog : public Animal {
public:
    std::string go(int n_times) override {
        std::string result;
        for (int i=0; i<n_times; ++i)
            result += bark() + " ";
        return result;
    }
    virtual std::string bark() { return "woof!"; }
};
"""

CPP_OUTPUT_STUB = """
#include "dog.hpp"
#include <pybind11/pybind11.h>
namespace py = pybind11;

// <litgen_glue_code>
// </litgen_glue_code>
void py_init_module_test_output(py::module& m) {
// <litgen_pydef>
// </litgen_pydef>
}

PYBIND11_MODULE(test_output, m) { py_init_module_test_output(m); }
"""

PYI_OUTPUT_STUB = """
# <litgen_stub>
# </litgen_stub>
"""

HEADER_FILES = ['dog.hpp']

if TRIGGER_THE_BUG:
    HEADER_FILES.append('animal.hpp')
    with open('animal.hpp', 'w') as fd:
        fd.write(ANIMAL_CLASS)

with open('dog.hpp', 'w') as fd:
    if TRIGGER_THE_BUG:
        fd.write('#include "./animal.hpp"\n')  # I had to add "./" here, so that it is handled by local_includes_startwith
    else:
        fd.write(ANIMAL_CLASS)
    fd.write(DOG_SUBCLASS)

with open('test_output.cpp', 'w') as fd:
    fd.write(CPP_OUTPUT_STUB)

with open('test_output.pyi', 'w') as fd:
    fd.write(PYI_OUTPUT_STUB)


HEADER_FILES = ['animal.hpp', 'dog.hpp']


#
# Using the amalgamated_header.py module from codemanip (provided by litgen)
#
from codemanip.amalgamated_header import AmalgamationOptions, write_amalgamate_header_file

amalgamation_options  = AmalgamationOptions()
# Base dir for the include paths
amalgamation_options.base_dir = '.'
# All possible subdirs for includes
amalgamation_options.include_subdirs = ["./"]
# Start of the local includes (so that we do not include <string> for example)
# In your example, I had to add "./" to the include path, so that it is handled by local_includes_startwith
amalgamation_options.local_includes_startwith = './'
# You wlll need a main header that includes all the API you want to expose
amalgamation_options.main_header_file = 'dog.hpp'
# Destination file for the amalgamated header
amalgamation_options.dst_amalgamated_header_file = 'amalgamated_header.hpp'
# Let's write the amalgamated header
write_amalgamate_header_file(amalgamation_options)




conf = LitgenOptions()
conf.class_override_virtual_methods_in_python__regex = '^Animal$|^Dog$'
write_generated_code_for_file(
    conf,
    #input_cpp_header_files=HEADER_FILES,
    input_cpp_header_file='amalgamated_header.hpp',
    output_cpp_pydef_file='test_output.cpp',
    output_stub_pyi_file='test_output.pyi',
)

[Dan]

Ah, thank you. I'd found a workaround parsing free functions and class members in separate passes (with parent classes inlined for the class members), but yours is better.

Could write_amalgamate_header_file be added to the online docs? I see a reference to it in the DasLib example but I didn't see an explanation of why that approach needed to be chosen over passing a list of files to write_generated_code_for_files.

[Pascal Thomet]

Could write_amalgamate_header_file be added to the online docs?

See new doc