How to define custom parameters in an interface data definition (*.capnp) ?

[Antoine Riard]

I've successfully added and built API/data definitions for my new Validation interface

    namespace interfaces {
    
    //! Inteface giving clients access to the validation engine.
    class Validation
    {
    public:
        virtual ~Validation() {}
    
        // Check if headers are valid.
        virtual bool validateHeaders(const BlockHeader& header) = 0;
    };
    
    
    } // namespace interfaces

    using Cxx = import "/capnp/c++.capnp";
    $Cxx.namespace("ipc::capnp::messages");
    
    using Proxy = import "/mp/proxy.capnp";
    $Proxy.include("interfaces/validation.h");
    $Proxy.include("ipc/capnp/validation.capnp.h");
    
    interface Validation $Proxy.wrap("interfaces::Validation") {
        destroy @0 (context :Proxy.Context) -> ();
        validateHeaders @1 (context :Proxy.Context, header: BlockHeader) -> (result: Bool);
    }

I would like to add a custom parameters to my interface :

struct BlockHeader $Proxy.wrap("BlockHeader")
{
    nVersion @0 :Int32;
    hashPrevBlock @1 :Data;
    hashMerkleRoot @2 :Data;
    nTime @3 :UInt32;
    nBits @4 :UInt32;
    nNonce @5 :UInt32;
}

But so far, I'm hitting libmultiprocess compilations errors which are beyond my understanding for
now.

AFAIU by observing the interfaces in #10102, you have two options to add a custom parameters :

• wrapping a struct duplicated in the API definition (e.g struct WalletTxOut $Proxy.wrap("interfaces::WalletTxOut"))
• declaring a custom reader/builder in "types files" linking API/data definitions (e.g CustomReadMessage/CustomBuildMessage)

I should rebase my branch on top of #10102, as this branch includes a lot more of common helpers.

Feel free to correct the terminology I used, I'm progressively catching up on libmultiprocess usage :)

[Ryan Ofsky]

I'm happy to help debug if you post a git tag. I think probably for your case, you don't need to build on bitcoin/bitcoin#10102 and just bitcoin/bitcoin#19160 should be sufficient.

Everything you posted and wrote is right. To use custom C++ types in interfaces, you need to either use $Proxy.wrap and list all fields in the type, or you need implement CustomReadField/CustomWriteField/CustomPassField overloads to read/write the C++ type as a capnp type.

The CustomReadMessage/CustomBuildMessage overloads you mentioned are added in bitcoin/bitcoin#10102. They are a simpler alternative to CustomReadField/CustomWriteField/CustomPassField and can be defined in .cpp instead of .h files since they aren't template functions. bitcoin/bitcoin#10102 also adds CustomReadField/CustomWriteField/CustomPassField overload that detect if a type has Serialize/Unserialize methods and will use those if them if they are available.

[Antoine Riard]

Thanks to propose to help on the debug, here the git tag : ariard/bitcoin@ddcfd07. Note, you need to pass the following configuration flag --enable-altnet on top of the --enable-multiprocess one.

I'm trying to follow the first path for my custom C++ types, i.e $Proxy.wrap and list all fields in the type. I'm encountering the following build error :

/home/user/Bitcoin/bitcoin/depends/x86_64-pc-linux-gnu/include/mp/proxy-types.h:942:30: error: no matching function for call to ‘BuildPrimitive(mp::InvokeContext&, const uint256&, mp::TypeList<capnp::Data::Builder>)’
     output.set(BuildPrimitive(invoke_context, std::forward<Value>(value), TypeList<decltype(output.get())>()));                                                                                                  
                ~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~       

AFAICT, such BuildPrimitive() is present in bitcoin/bitcoin#10102 to build the node interface custom types (in src/ipc/capnp/node-types.h). Though it's just returning a ::capnp::Void, I'll try to do the same for my Validation interface.

[Ryan Ofsky]

The BuildPrimitive error indicates that no Build overload is present that can convert const uint256& C++ values to capnp::Data values. (Sorry about the confusing error message. I want to improve these errors by collapsing Build* overloads with C++17 if constexpr expressions in #46. This error happens because BuildPrimitive is the lowest priority build function that gets called when no other build functions can be applied the data type).

Because uint256 is a serializeable type, the fix for this is to supply a CustomBuildField function that can serialize it, like:
https://github.com/ryanofsky/bitcoin/blob/db776ff2e92eb51fd51f56f9557546d4484a44d9/src/ipc/capnp/common-types.h#L248-L262 added in bitcoin/bitcoin#10102

I'm building your branch and can post a patch soon.

[Ryan Ofsky]

Following commit should fix the BuildPrimitive error: ryanofsky/bitcoin@e53c7a9. It just pulls in common-types.h functions from bitcoin/bitcoin#10102 to be able to handle serializable types.

There are still some other build errors on the branch, which I can try to help with, so feel free to ask any followups!

[Antoine Riard]

Thanks for the answer on this, I finally fixed the other build errors by rebasing and tweaking few others files. I can build the data definitions I want and making progress on my altnet branch.

I don't have other libmultiprocess questions for now :)

[Ryan Ofsky]

I don't have other libmultiprocess questions for now :)

Will close this issue, but feel free to reopen or post questions or work-in-progress code in new issues. Happy to help!

[Antoine Riard]

Sure, fyi the WIP branch is here: https://github.com/ariard/bitcoin/commits/2021-07-altnet-lightning, where I'm able to chain multiple processes : bitcoin-node -> bitcoin-altnet -> altnet-lightning but give me few more days to rebase/cleanup code/test with lightning driver then I think I'll have more advanced questions on the code generator itself!