RFC: Miden SDK Outline

Question

RFC: Miden SDK Outline

bitwalker opened this issue 10 months ago · comments

Paul Schoenfelder commented 10 months ago

This issue is just an initial sketch of my proposal for how to approach development of the Miden SDK tooling:

Determine whether WebAssembly Interface Types (WIT) is a good solution for specifying function types in Miden. ¹
Set up the Cargo extension to compile two variants of a crate, one which is compiled to wasm32-unknown-unknown (last), and the other which is compiled to the host target, both as dylibs. ²
As a short-term hack that will let us experiment with this approach as a solution, rather than add code to the compiler to work with the WIT spec, we will consume it directly in the Cargo extension to emit Rust code that will be compiled along with the rest of the user-written code in the crate for the wasm32-unknown-unknown target. ³
Finally, implement support in the compiler's Wasm frontend, to expect a WIT specification to be provided along with the Wasm module, which will be used to enrich the function signatures in the IR type system when translating from Wasm to Miden IR. ⁴

I'm about to crash here, so this is basically just flushing this stuff out of my brain, hopefully its useful to you @greenhat until I get a chance to clean it up a bit and expand on some of these points.

A good solution in this case means that it can be used to represent high-level Rust (and other language) types using a common format (preferably in a way that lets us map those high-level types to our IR type system), which provides us the ability to query the types, sizes, and alignment of arguments and return values for a given function. Additionally, we want to be able to generate code to construct this specification in-memory, as well as parse/load it from a textual format, and use the in-memory spec to query things such as the number of parameters, what type are those parameters, what are the size and alignment, etc. ↩
Using conditional compilation, the latter target will contain a function which, when loaded dynamically and called, will return the WIT specification for the account/note functions in that crate. That spec will be used by the compiler to "enrich" the wasm32-unknown-unknown artifact with type information from the original program, enabling the compiler to handle the calling convention details. ↩
The generated code can simply be placed in a build directory for the crate and included; or placed in the source tree, either approach works. The Rust code we're generating here are wrappers which handle the gritty details of the calling convention (a wrapper for each public contract function defined in the crate, which delegates to the actual function after decoding the arguments using the calling convention rules; and a wrapper for external contract functions, which encodes the high-level Rust types using the calling convention rules, and then invoking the external function) ↩
Simultaneously, we'll implement the calling convention handling for call in the compiler backend. Once complete, we can rip out the hack from the previous stage ↩

Denys Zadorozhnyi · Answer 1 · Wed Nov 29 2023 20:24:13 GMT+0800 (China Standard Time)

From what I gathered so far, the Wasm component model ecosystem is built around the idea of treating the WIT as a "source of truth" for the component's interface and generating bindings for the various languages from it. The only project I found that tries to generate WIT file from a Rust code is an abandoned https://github.com/bnjjj/witgen. So it seems like the WIT generation will not be a "walk in the park" task.
Regarding the WIT format fitting. Although the WIT format itself is highly abstract there is https://github.com/WebAssembly/component-model/blob/main/design/mvp/Explainer.md#canonical-abi which describes how these types are "lowered" to the core Wasm types in https://github.com/WebAssembly/component-model/blob/main/design/mvp/CanonicalABI.md#flat-lowering so it should help in determining layout and sizes for a type defined in WIT.

During my exploration, I quickly glanced through the Wasm component code built from the Rust "reactor" Wasm component template and found that Wasm binary contains a lot of extra info (including the whole WIT) compared to the core Wasm binary. For example, here is the excerpt of the definition for the hello_world function:

  (type (;32;) (func (result string)))
  (alias core export 2 "hello-world" (core func (;49;)))
  (alias core export 2 "cabi_post_hello-world" (core func (;50;)))
  (func (;17;) (type 32) (canon lift (core func 49) (memory 0) string-encoding=utf8 (post-return 50)))
  (export (;18;) "hello-world" (func 17))

compiled from the following Rust code:

cargo_component_bindings::generate!();

use bindings::Guest;

struct Component;

impl Guest for Component {
   /// Say hello!
   fn hello_world() -> String {
       "Hello, World!".to_string()
   }
}

with the following WIT:

package component:wit-lib;

/// An example world for the component to target.
world example {
    export hello-world: func() -> string;
}

I believe it is worth further exploring what's inside the Wasm component binary.

Paul Schoenfelder · Answer 2 · Thu Nov 30 2023 15:52:08 GMT+0800 (China Standard Time)

After doing some more investigation today and writing notes as I went along, I think I have a better idea of what we can/can't do along these lines. The following are notes I wrote up, let's talk around the same time tomorrow that we did today and see where we end up.

Either require manually authoring .wit files, where each account/note/transaction script are defined in their own component; OR automate this process (see the _Expressing Accounts, Notes, and Transaction Scripts_ section below)
The wit-parser crate provides an in-memory representation of WIT packages, parsing, as well as the ability to convert a WIT type into the flattened core Wasm type representation as described in the Component Model "Canonical ABI" document.
The wit-component crate provides pretty-printing of WIT packages (using types from the wit-parser crate), as well as the ability to decode a Wasm module and extract the WIT information from it (if it was compiled as a component model module). It has some other features as well, but those listed are the most interesting for us.
The wit-bindgen-core crate provides low-level traits/utilities for implementing your own bindings between native Wasm functions and WIT functions. Probably not a path we'll go down, but worth knowing.
The wit-bindgen crate can be used directly rather than working through cargo-component, if we don't want all of the opinions baked into the latter
The wasmtime crate implements the component model as a Wasm host, but the interesting bit for is the Lift and Lower traits. As an aside, maybe we should consider making this a more intrinsic part of the Miden VM itself, i.e. requiring MASM modules to have WIT specifications, and have Miden implement the component layer, but not sure if that'll fly with the team or not - it does solve some problems in a nice way though for sure.
The witgen crate is not something we'd want to use directly, but the approach it takes would be an option if we determine we want to generate WIT from Rust sources, rather than the take the approach of manually-authored WIT
The wasm-tools crate/tools provides some handy tooling for working with WIT/Wasm modules

Expressing Accounts, Notes, and Transaction Scripts

Let's assume that we require users to specify the ABI of their contracts using WIT, building on the native WIT types, in addition to WIT packages provided by the Miden SDK. That might look something like:

// <workspace>/wit/my-wallet.wit
package miden:my-wallet@0.1.0;

use miden:base/types@1.0.0;
use miden:base/account@1.0.0;

interface my-wallet {
    use types.{asset, tag, recipient};

    receive-asset: func(asset: asset);
    send-asset: func(asset: asset, tag: tag, recipient: recipient);
}

world abi {
    import account.{add-asset, remove-asset};
    export my-wallet
}

Our miden:base package might look something like this:

package miden:base@1.0.0;

interface types {
    type felt = u64;
    type word = tuple<felt, felt, felt, felt>;
    
    type account-id = felt;
    type recipient = word;
    type tag = felt;
    record fungible-asset {
        asset: account-id,
        amount: u64
    }
    type non-fungible-asset = word;
    variant asset {
        fungible(fungible-asset),
        non-fungible(non-fungible-asset),
    }
}

interface account {
    use types.{account-id, asset};

    add-asset: func(asset: asset) -> asset;
    remove-asset: func(asset: asset) -> asset;
}

interface tx {
    use types.{asset, tag, recipient};

    create-note: func(asset: asset, tag: tag, recipient: recipient);
}

In Rust, the user would then need to write code like the following (if we assume no additional infrastructure is in play, i.e. proc macros), and we assume we're using cargo-component for the moment:

# Cargo.toml
[package]
name = "my-wallet"
version = "0.1.0"
edition = "2021"

[dependencies]
cargo-component-bindings = "0.5"

[lib]
crate-type = ["cdylib"]

[package.metadata.component]
package = "miden:my-wallet"

[package.metadata.component.target]
path = "wit/my-wallet.wit"
world = "abi"

// src/lib.rs

cargo_component_bindings::generate!();

use bindings::exports::miden::abi::my-wallet::{Guest, Asset, Tag, Recipient};

// Bring imports into scope
use bindings::miden::abi::{account, tx};

// Implement the interface expressed in my-wallet.wit
struct MyWallet;
impl Guest for MyWallet {
	fn receive_asset(asset: Asset) { account::add_asset(asset); }
    fn send_asset(asset: Asset, tag: Tag, recipient: Recipient) { 
        let asset = account::remove_asset(asset);
        tx::create_note(asset, tag, recipient);
    }
}

With what is described above, we have the following:

A specification of the externally-callable items in the my-wallet library in terms of high-level types
Auto-generated bindings for the MyWallet account, as well as bindings for functions provided by Miden Base that aren't implemented in Rust
A WIT package that can be published and consumed as a dependency from other Miden-based projects

However, there are still additional problems that are not solved yet:

This does not tell us which functions are called via call vs exec. We don't want to assume that every function in a WIT interface is call'd, since that is not necessarily true. We need some additional proc macro machinery to emit that metadata somewhere as well so the compiler can see it. A rather simple approach for this would be to embed it in the crate itself, so that we can load the library dynamically and read the embedded metadata. It could be provided out-of-band as well, but since we can extract all of the WIT metadata from a Wasm module, it would be ideal if we could extract our own metadata as well to simplify things.
The actual lifting/lowering of arguments/results is hidden from us here, we would need to provide our implementation of the Bindgen trait from wit-bindgen-core in order to inject our own calling convention handling for calls (and in the prologue of the callee) in Rust. We could probably reuse the bulk of the implementation in the wit-bindgen-rust crate, and modify it as needed.

It occurs to me that all of this is really designed to have the host (i.e. the Miden VM) handle some of these details. For example, the host knows what the actual signatures of caller and callee are, is able to validate them, and also handles lifting/lowering to its own ABI. So in our case, all of the details of the calling convention that we're concerned about with regard to contract calls is something that would be handled transparently by the host in the Wasm component model view of things.

Bobbin Threadbare · Answer 3 · Thu Nov 30 2023 17:34:20 GMT+0800 (China Standard Time)

Having just skimmed through the discussion above, a couple of random comments:

This does not tell us which functions are called via call vs exec. We don't want to assume that every function in a WIT interface is call'd, since that is not necessarily true.

Not sure if this helps, but for the purposes of Miden rollup we can make the following assumptions:

All functions exposed publicly by the account should be invoked via the call instruction.
Everything else should be invoked via the exec instruction.

interface types {
    type felt = u64;
    type word = tuple<felt, felt, felt, felt>;
    
    type account-id = felt;
    type recipient = word;
    type tag = felt;
    record fungible-asset {
        asset: account-id,
        amount: u64
    }
    type non-fungible-asset = word;
    variant asset {
        fungible(fungible-asset),
        non-fungible(non-fungible-asset),
    }
}

Defining Asset type would be a bit more tricky (I actually don't know what's the best way to do it in Rust either). The main reason is that Asset is just a Word. The most significant bit in this word specifies whether this is a fungible or non-fungible asset. Then, the structure of the word depends on the type, specifically:

Fungible assets have the following form: [amount, 0, 0, account_id], where both account_id and amount are Felts.
Non-fungible assets have the following form: [d0, d1, account_id, d3] where d0, d1, d3 are field elements such that d3's most significant bit is always set to 0.

We could probably change the encoding scheme slightly, but it would be important to keep both fungible and non-fungible assets encoded as exactly one word.

// /wit/my-wallet.wit
package miden:my-wallet@0.1.0;

use miden:base/types@1.0.0;
use miden:base/account@1.0.0;

interface my-wallet {
use types.{asset, tag, recipient};
receive-asset: func(asset: asset);
send-asset: func(asset: asset, tag: tag, recipient: recipient);
}

world abi {
import account.{add-asset, remove-asset};
export my-wallet
}

Would we be able to attach metadata to functions in definitions above? Specifically, we may need to specify MAST roots for receive-asset and send-asset functions so that they can be invoked from note/tx scripts.

Denys Zadorozhnyi · Answer 4 · Thu Nov 30 2023 18:25:18 GMT+0800 (China Standard Time)

However, there are still additional problems that are not solved yet:

This does not tell us which functions are called via call vs exec. We don't want to assume that every function in a WIT interface is call'd, since that is not necessarily true.

I believe we can treat every public method of the account interface as call-able. We need a way of recognizing a component as an account. We can require that every account component implements a special interface. For example, it can contain a function that returns the account human-readable name - account_name:

interface Account {
    account_name() -> string;
}

It occurs to me that all of this is really designed to have the host (i.e. the Miden VM) handle some of these details. For example, the host knows what the actual signatures of caller and callee are, is able to validate them, and also handles lifting/lowering to its own ABI. So in our case, all of the details of the calling convention that we're concerned about with regard to contract calls is something that would be handled transparently by the host in the Wasm component model view of things.

This is such a great point!

Overall, I'm increasingly convinced that embracing the Wasm component model will spare us from writing a lot of almost-the-same-but-different type of stuff and open Miden VM and rollup to the wider Wasm ecosystem.

I'm eager to implement our basic wallet example as components and check the compiled Wasm! I see the account code as a reactor(lib) component exporting an interface and note with tx scripts as cmd components using it. With the Miden SDK as a reactor component.

Paul Schoenfelder · Answer 5 · Fri Dec 01 2023 04:45:30 GMT+0800 (China Standard Time)

Not sure if this helps, but for the purposes of Miden rollup we can make the following assumptions:

All functions exposed publicly by the account should be invoked via the call instruction.

Everything else should be invoked via the exec instruction.

The main issue with this is you might want to publish components/libraries containing types/helpers beyond just the account and it's interface. For example, the miden:base/types interface in the example .wit of my comment defines a variety of types that are used in the implementation of Miden-related things (such as accounts), and the miden:base/account interface exposes some static functions (that would map to things implemented in Miden Assembly presumably). We'd still want to allow expressing that stuff in WIT so that bindings can be auto-generated for those things.

I don't think this is a issue in practice though, we can use doc attributes in the WIT files, something like the following:

/// The MyWallet account type
/// @account
interface my-wallet {
    /// A type used in MyWallet APIs
    type foo = u32;

    /// Send an asset to MyWallet
    /// @contract
    send-asset: func(...);

    /// Receive an asset to MyWallet
    /// @contract
    receive-asset: func(...);

    /// An example helper function that doesn't interact with MyWallet directly
    create-foo: func(...) -> foo;
}

In this case, we can use the @account and @contract attributes to enrich the metadata contained in the WIT to provide us additional information in tooling that consumes the WIT (compiler, perhaps the VM, etc.).

Defining Asset type would be a bit more tricky (I actually don't know what's the best way to do it in Rust either). The main reason is that Asset is just a Word. The most significant bit in this word specifies whether this is a fungible or non-fungible asset. Then, the structure of the word depends on the type, specifically:

As an aside: I'm a bit unclear on what "most significant bit" means here - my understanding is that field elements have no bitwise representation, unless they are being used to represent a u32, but that doesn't sound like the case here. Is the code that handles encoding asset values assuming the representation of the field element as something like a 63-bit unsigned integer?

In any case, the thing to keep in mind is that the way a type is encoded in the guest language (e.g. Rust, Miden Assembly, whatever), is completely up to the language. What WIT is specifying is the "Canonical ABI", i.e. the common representation which any language can lift into their own types as they see fit. There are some caveats there - generated bindings probably aren't going to do any clever niche optimizations and things of that nature; but you aren't required to generate bindings either. The specific case of Asset as you've described is complicated by the fact that WIT doesn't have two things we'd need to natively represent it: 1.) we can't specify the type used for the variant tag (e.g. bool), and 2.) we don't have arbitrary precision integers, e.g. u63. Obviously we can represent Asset in WIT using a less natural representation, or by making the type opaque and using free functions to access information about it.

In practice though, we will have a Rust crate that provides a facade for many of the Miden Assembly intrinsics anyway, so the WIT for the intrinsics could use opaque types (e.g. word), and the Rust crate would handle converting to that opaque representation when delegating to the intrinsics, but otherwise could use the nicer representation everywhere else.

We could probably change the encoding scheme slightly, but it would be important to keep both fungible and non-fungible assets encoded as exactly one word.

This much we can certainly enforce in WIT (the size), but depending on how many bits of the various fields are actually unused, we may not be able to find a nicer way to pack things. Ultimately though, as I mentioned above, we can keep the single-word encoding for use internally, but present the nicer encoding to the world, and simply translate to the internal encoding as needed.

Would we be able to attach metadata to functions in definitions above? Specifically, we may need to specify MAST roots for receive-asset and send-asset functions so that they can be invoked from note/tx scripts.

We can, because doc comments are preserved through all layers of the stack, so we could decorate items with things like /// @mast 0x... - that said, I think we would want that specific metadata to live separately.

We should set up a call to discuss packaging in general, but after spending quite a bit of time digging through all of this, I think WIT should/will play a major role here. Just to give you a rough sense of what I've been thinking about/got in mind here, the following is a description of what I think a "package" in Miden would look like/contain:

A WIT package (one or more .wit files) describing the component. This can be in binary form or text, either encoding is fine, because you can get the text from the binary and vice versa.
A manifest file which contains the MAST root hash for every function exported by the component and its dependencies.
The compiled MAST code (which is presumed to contain all of the hashes in the manifest above, and could be validated to guarantee that)
Source maps/debug info (optional, but provides some nice benefits for downstream users of the package)

This could then be published somewhere, perhaps to a Miden-specific registry of some kind, but multiple sources could easily be supported. If you want to then write a note script involving a specific account type, for which a package such as I just described has been published, the process is simple:

Create a new Rust project, e.g. cargo miden create-note or whatever
In the Cargo.toml, specify the dependency and its version (which could be a hash rather than a semver version, but we could also provide a way to map semver versions to hashes if we're designing the registry). That would look something like:

[package.metadata.miden.dependencies]
my-wallet = { hash = "0x..." }

Run cargo miden get to fetch the configured dependencies and unpack them so they can be used to generate bindings
Write a .wit file describing the note and importing dependencies as needed:

package miden:my-note;

default world note {
    use miden:my-wallet/types.{wallet};
    export run: func(wallet: wallet);
}

Then generate the bindings in src/lib.rs and implement the run function:

cargo_component_bindings::generate!();

use bindings::exports::miden::my_note::note::{Guest, Wallet};

struct MyNote;
impl Guest for MyNote {
    fn run(wallet: Wallet) { todo!() }
}

Obviously, a few things aren't specified here, but the general workflow is pretty close. The nice thing, is that because our tooling would understand the package format, all of this works pretty seamlessly. A user could even test/debug their whole stack by having the Cargo extension compile and load their script, along with the MAST from all dependencies, into a test harness or debugger. If source maps are present, we'd even be able to give them useful traces, and show sources when stepping through the code (assuming the sources were provided some way). If the VM itself understands the package format, it can even lean on existing tools for linking components without having to reimplement that from scratch.

I guess an open question there is where to publish such an artifact. It feels to me like the natural place to store it is in fact the chain itself (i.e. I'm assuming we're not going to store just MAST files on-chain, presumably some metadata is needed, and this package format would be suitable for both on-chain storage as well as off-chain storage).

By storing it on-chain, you also gain the ability to extract the WIT interface for any contract, the hashes of the functions, etc., making introspection tools much easier to build. Another consequence of doing so is that the chain itself is a registry of sorts, i.e. when a new contract is published, all of the data associated with it is published at the same time, and you now have a permalink to the specific version of the contract and its metadata. Of course, we'd still need versioning for the packaging format itself, but that is unlikely to change in backwards-incompatible ways very frequently.

For off-chain storage (and perhaps even on-chain storage), a lot of the core registry tooling has already been implemented via warg, though I haven't looked at it to see how opinionated it is, but we could in theory reuse a lot of the work done there.

Anyway, this is a big brain dump, but I'd like to discuss some of this soon, because in particular I think having the VM be a more intrinsic part of this would provide some significant wins across the board, but I'd like to work through the details of what problems we'd be aiming to solve, and what all would be required in the VM itself.

Paul Schoenfelder · Answer 6 · Fri Dec 01 2023 05:26:23 GMT+0800 (China Standard Time)

I believe we can treat every public method of the account interface as call-able.

We can do this yes, but I'd prefer a more explicit approach if possible, such as decorating specific functions with a marker attribute of some kind. That said, using a specific interface that must be implemented explicitly might be enough, we should explore that further.

Overall, I'm increasingly convinced that embracing the Wasm component model will spare us from writing a lot of almost-the-same-but-different type of stuff and open Miden VM and rollup to the wider Wasm ecosystem.

Definitely, I've got the same feeling, and once we get some of these unspecified details pinned down, I think we'll have a very clear path to shipping an MVP of all this stuff!

I'm eager to implement our basic wallet example as components and check the compiled Wasm! I see the account code as a reactor(lib) component exporting an interface and note with tx scripts as cmd components using it. With the Miden SDK as a reactor component.

That sounds more or less correct to me, I think a big question is going to be how we express certain core interfaces and types, but we can play with that to see what produces the most ergonomic APIs.

One thing I'm very keen to investigate further are resources. They map very closely to the concept of resources in Move actually, and using some of the features of the component model, we can do some nice things with them. To summarize: WIT has the concept of resource types, which are associated with handles that can be owned or borrowed. Resources are tracked in the component model, meaning that a borrowed resource must always be released at the end of the function borrowing it, you cannot borrow an owned resource, etc. The following is an example of what things might look like if we represented assets as resources (ignore the functionality provided, its just to illustrate what the code looks like):

package miden:base;

interface assets {
    resource fungible-asset {
        constructor(amount: u64);
        amount: func() -> u64;
        split: func(amount: u64) -> result<fungible-asset, _>;
        destroy: static func(%self: own<fungible-asset>);
    }

    variant asset {
        // This variant holds an owning handle to a `fungible-asset`
        fungible(own<fungible-asset>),
    }
}

world sdk {
    export assets;
}

This would be implemented by the following Rust code:

wit_bindgen::generate!({
    world: "sdk",
    exports: {
        "miden:base/assets/fungible-asset": FungibleAsset
    }
});

use core::cell::RefCell;

pub struct FungibleAsset {
    account_id: AccountId,
    amount: RefCell<u64>,
}

impl exports::miden::base::assets::GuestFungibleAsset for FungibleAsset {
    fn new(amount: u64) -> Self {
        let account_id = todo!();
        Self { account_id, amount: RefCell::new(amount) }
    }

    fn amount(&self) -> u64 { *self.amount.borrow() }

    fn split(&self, amount: u64) -> Result<Self, ()> {
        let current_amount = self.amount.borrow_mut();
        if current_amount < amount {
            return Err(());
        }
        current_amount -= amount;
        Ok(Self::new(amount))
    }

    fn destroy(mut this: exports::miden::base::assets::OwnFungibleAsset) {
        // ..snip..
        *this.amount.borrow_mut() = 0;
    }
}

I don't know if there will be a place where it makes sense to use resources, but its something we should keep in mind.

Bobbin Threadbare · Answer 7 · Fri Dec 01 2023 18:59:14 GMT+0800 (China Standard Time)

As an aside: I'm a bit unclear on what "most significant bit" means here - my understanding is that field elements have no bitwise representation, unless they are being used to represent a u32, but that doesn't sound like the case here. Is the code that handles encoding asset values assuming the representation of the field element as something like a 63-bit unsigned integer?

It is possible to convert field elements into an integer representation, and one can define different rules for how to do so. For example, we could represent our field elements as unsigned 64-bit integers, but we could also represent them as signed 64-bit integers (important to note that every field element would map uniquely to a 64-bit integer, but not every 64-bit integer would map uniquely to a field element).

To clarify my statement, I meant the most significant bit if we interpret a filed element as an unsigned 64-bit integer.

In practice though, we will have a Rust crate that provides a facade for many of the Miden Assembly intrinsics anyway, so the WIT for the intrinsics could use opaque types (e.g. word), and the Rust crate would handle converting to that opaque representation when delegating to the intrinsics, but otherwise could use the nicer representation everywhere else.

Yep, I think we can figure out what to do once we start implementing. btw, once of the reasons for using words is that they are pretty efficient to work with in the VM (e.g., storing words to/from memory can be done with very few instructions).

4. Write a .wit file describing the note and importing dependencies as needed:

Why would we need to write a .wit file for a note? I was thinking .wit files would be needed for accounts only as accounts expose public interfaces which something could use downstream. Notes don't really have such interfaces - i.e., there is nothing that would need to import a note.

5. Then generate the bindings in src/lib.rs and implement the run function:

cargo_component_bindings::generate!();

use bindings::exports::miden::my_note::note::{Guest, Wallet};

struct MyNote;
impl Guest for MyNote {
    fn run(wallet: Wallet) { todo!() }
}

I also didn't quite follow this part. I was thinking it could look something like this:

use some_crate::Wallet;

struct MyNote;
impl MyNote {
    fn run(wallet: Wallet) { todo!() }
}

Where some_crate is a package imported in the usual way by specifying a dependency in Cargo.toml.

I guess an open question there is where to publish such an artifact. It feels to me like the natural place to store it is in fact the chain itself (i.e. I'm assuming we're not going to store just MAST files on-chain, presumably some metadata is needed, and this package format would be suitable for both on-chain storage as well as off-chain storage).

On-chain storage is one of the most scarce resources - so, we'd want to compress things as much as possible. This means that the amount of metadata we'd include would be minimal. In fact, in many cases we'd probably skip putting most of the actual code on-chain as well. The nice thing about MAST is that we can selectively remove "commonly known" code without changing anything about program semantics. In the most extreme case, we'd put only the MAST root of a program (i.e., 32 bytes) on chain - and if this program is "well known" this would be sufficient for people to execute it.

In general, I think the model we should go for is that developers will need to obtain artifacts from some off-chain sources to be able to develop against published smart contracts.

Paul Schoenfelder · Answer 8 · Sat Dec 02 2023 03:02:20 GMT+0800 (China Standard Time)

To clarify my statement, I meant the most significant bit if we interpret a filed element as an unsigned 64-bit integer.

Ah I see, I was aware that we could technically make assumptions like that (i.e. that a 63-bit integer could be losslessly converted to a field element and vice versa), but I actually didn't realize that the 64th bit would be anything other than zero when mapping unsigned integers to the field (e.g. Felt::new(1 << 63).as_int() would be lossy because the value is larger than the field modulus). In order to access that bit then, I'm guessing we're splitting the felt into two u32 limbs using u32split, and then accessing the most significant bit of the most significant limb? I suppose I should just go take a look at the code that implements this rather than asking here 😅.

Yep, I think we can figure out what to do once we start implementing. btw, once of the reasons for using words is that they are pretty efficient to work with in the VM (e.g., storing words to/from memory can be done with very few instructions).

For sure, though I think in practice what you'd do in Rust specifically is pass around an asset by reference, which is even more efficient, and load/store specific fields as needed. That's annoying to do in hand-written MASM though, so having a packed encoding there makes a lot of sense. In any case, having both a packed and expanded representation is likely to happen anyway, since developers are going to want to work with a more ergonomic representation in their own code. I think it makes sense for us to expose two sets of APIs: the lower-level one that is expressed in terms of the packed representation, and a wrapper around the low-level APIs that uses the expanded representation.

Why would we need to write a .wit file for a note? I was thinking .wit files would be needed for accounts only as accounts expose public interfaces which something could use downstream. Notes don't really have such interfaces - i.e., there is nothing that would need to import a note.

It isn't to support downstream use of the note's API, it's about defining the note as a component. To expand on that a bit, the main reasons to require a WIT package for each project type are:

It is used to define a component in the component model. At a minimum this provides rich type signatures in the compiled Wasm that we can use in the compiler. For example, it would be possible to express the expected input types for the note this way. Compiling from Rust to Wasm without the component model does not provide us with any meaningful type information (everything is essentially expressed in terms of the Wasm i32 type). This results in much less efficient code being generated.
It would allow us to require that an account/note/tx script implements some interface, regardless of what language it is defined in. ¹
It would allow us to support defining accounts/notes/tx scripts in a single crate. ²
It drives generation of bindings. ³

The footnotes further expand on those points, but the long and short of it is that defining a note as a WIT package is just as beneficial as doing so for an account, perhaps even more so, but either way is critical in actually delivering on some of the big wins of the component model.

On-chain storage is one of the most scarce resources - so, we'd want to compress things as much as possible.
..snip..
In general, I think the model we should go for is that developers will need to obtain artifacts from some off-chain sources to be able to develop against published smart contracts.

Makes sense, it is a bit of a shame that we don't get the "permanence" of publishing the package to the chain itself, but understandable why that wouldn't be desirable from a storage point of view. I am curious what the process of actually publishing a package (including the on-chain parts) would look like, particularly with identifying which parts of the code can be elided on chain, but I don't think it changes much from the actual packaging point of view (since presumably whatever publishes code on-chain will extract the package and post-process it in some way).

For off-chain solutions, we have some great off-the-shelf tools we can build on, such as warg, which would allow us to support multiple sources (path dependencies, source control, an actual registry if we choose to maintain an official one, etc.).

In Rust, we could do something like require a note to implement a specific marker trait, but that on its own doesn't do anything useful for us because we aren't actually consuming the note in Rust, we're compiling to Wasm and consuming that. At a minimum we need metadata written/generated somewhere that tells us what is in that Wasm module (is it a note/account, which functions are part of the note/account interface, etc.). We can do that with proc macros (though we would then need to determine what format to use, where to store it, etc.), but that leaves us with a Rust-specific solution that we have to re-invent for every language that wants to compile to Miden and interact with the rollup. By requiring a WIT description for each type of rollup component (accounts/notes/tx scripts), even if it is extremely minimal, we not only solve all of those issues, we essentially solve it for all languages simultaneously. We don't have to have a lot of complex proc macro machinery, or try to replicate that in another language with less sophisticated code generation tools, we just need a WIT package that describes the code we're given. In the case of Wasm modules in particular, we're guaranteed that the module matches the WIT description of it (which is embedded in the module itself), because otherwise the module would not have passed validation. ↩
A WIT file can contain multiple "worlds", and each world represents a concrete component, so for example, a crate with an account and a couple re-usable notes, would be expressed in WIT as three worlds that each implement a specific interface (either account or note). We can use this metadata to process a module containing those items in such a way that we maximize code reuse (i.e. the account or note-specific code could be extracted to separate MASM modules, which depend on a single core module containing all of the library code shared between them). ↩
We want to avoid linking dependencies (such as the account a note depends on) into the compiled artifact. If you import those dependencies in Rust the usual way, any referenced code gets pulled in to the current crate's codegen unit. The main problem with this is that this results in a lot of code duplication, but it also will not interact well with how notes are intended to call accounts, because from the Rust compiler's perspective, if you are calling a function on an account, it's going to think you want to call that exact function, and may try to inline the call, amongst other things. So you have to break that link somehow. One way to do so is to define a crate which exposes only stubs, and have end users of your account depend on that crate. That crate would need to be hand written, and would be fragile. By defining a WIT package for the note however, and importing dependencies there, you can generate only the bindings you actually need, and ensure that you aren't unintentionally pulling in upstream code into your project. From the account author's perspective, they do not need to maintain a Rust crate so that Rust developers can use their account API - they just publish the WIT package, and the compiled code, and downstream users will have everything they need. ↩

Paul Schoenfelder · Answer 9 · Sat Dec 02 2023 11:01:40 GMT+0800 (China Standard Time)

@greenhat Some interesting things I found today that may be useful to you. Let's assume we have two components, foo and bar, in two separate crates:

// foo/wit/world.wit

package component:foo;

interface types {
    type id = u32;
    type word = tuple<u64, u64, u64, u64>;

    record foo {
        id: id,
        digest: word,
    }

    create-foo: func() -> foo;
}

world foo {
    export types;
}

// bar/wit/world.wit

package component:bar;

interface types {
    use component:foo/types.{id, foo};

    record bar {
        id: id,
        wrapped: foo
    }

    create-bar: func() -> bar;
}

world bar {
    import component:foo/types;
    export types;
}

Compiling bar to Wasm (as a component module), would produce something like the following (in Wasm text format, unimportant bits elided):

(component
    ; define the type of the `component:foo/types` interface
    (type
      (instance 
        ..snip..
        (export (;0;) "create-foo" (func (type ..))))
      )

    ; import an instance of the "component:foo/types" interface using the type above
    (import "component:foo/types" (instance (type 0)))

    ; define the core module containing the `create-bar` implementation
    (core module $bar_module
        (type (func (result i32)))
        (import "component:foo/types" "create-foo" (func $create_foo_import (type 0)))
        (func $component:bar/types#create-bar (type 0) (result i32)
            call $create_foo_import
        )
        (table 1 1 funcref)
        (memory 17)
        (export "memory" (memory 0))
        (export "component:bar/types#create-bar" (func $component:bar/types#create-bar))
    )

    ; import the `create-foo` function from the `component:foo/types` instance
    (alias export 0 "create-foo" (func (;0;)))
    
    ; synthesize a function to lower `create-foo` from the Canonical ABI so it may be called 
    ; from Core WebAssembly modules in this component
    (core func $create_foo_wrapper (canon lower (func 0)))

    ; define an anonymous instance that binds exports the wrapper as `create-foo`
    (core instance $binder
      (export "create-foo" (func $create_foo_wrapper))
    )

    ; instantiate $bar_module, using $binder to fulfill its dependency on `create-foo`
    (core instance $bar_instance (instantiate $bar_module
        (with "component:foo/types" (instance $binder))
      )
    )

    ; re-export the $bar_module exports
    (alias core export 1 "memory" (core memory (;0;)))
    (alias export 0 "foo" (type (;1;)))

    ; import the core `create-bar` function from $bar_instance
    (alias core export $bar_instance "component:bar/types#create-bar" (core func (;2;)))

    ; synthesize a function to lift the `create-bar` core function into the canonical ABI (for use by other components)
    (func $create_bar (type ..) (canon lift (core func 2)))

    ; define the `component:bar/example` component
    (component $example
      (type "import-type-bar" (type ..))
      (import "import-func-create-bar" (func (type ..)))
      (export "bar" (type ..))
      (export "create-bar" (func 0) (func (type ..))))

    ; instantiate the `component:bar/example` component
    (instance $example_instance (instantiate $example
        (with "import-func-create-bar" (func $create_bar))
        (with "import-type-bar" (type ..))
      )
    )

    ; use the `component:bar/example` instance to satisfy the `component:bar/types` interface
    (export "component:bar/types" (instance $example_instance))
)

So one thing to note about the above description is that actually describes how to build the component:bar/example component using what is present in the compiled artifact, rather than presenting a immutable description of the component. This is due to how the component model is intended to function, and actually works quite nicely for us.

In particular, note how the imported create-foo function, and exported create-bar function of the final component are "wrapped" by (canon lower ..) and (canon lift ..) respectively. The (canon lift ..) tells us that to call the function, we must lower arguments to the flattened core representation, and lift results into the Canonical ABI representation. The (canon lower ..) is essentially the inverse - we must lift arguments to the Canonical ABI representation, and lower results to the flattened core representation. The canon lift wrapper is used to expose core functions as component functions for other components, and canon lower is used to lower component functions into a form callable from core Wasm code.

So if you look at the definition of $bar_module, in particular the (import "component:foo/types" "create-foo" (func ..)) line. What's actually being imported is the (canon lower)'d version of create-foo from the component:foo/types component. In other words, code within $bar_module is calling a regular Wasm function that, when called, will "lift" its arguments into the Canonical ABI, and upon return, will lower out of it. On the flip side, the version of create_bar exported from the instantiated component is "lifted", so when that function is called, it should lower arguments from their Canonical ABI form, and lift results into it.

I found all of that quite interesting, as I think it provides us quite a bit of opportunity to actually decide how to compile these modules.

Reading through some of the design choices of the component model - particularly the emphasis on "shared-nothing" architecture I'm increasingly convinced that this is a really good fit for what we're doing in Miden. Because of the component model, it would actually be feasible for us to allow passing resources which reference memory in the caller's context to the callee without exposing the caller's context, since only the resource itself can access it, even if the resource leaked an address to its memory on purpose, it can't be used to access that memory, because the address is meaningless from any component other than the one defining the resource. And because the Canonical ABI requires that everything is passed by-value (except resources), there isn't any way to unintentionally pass pointers across contexts. In short, our calling convention rules are effectively enforced by the Canonical ABI already, all we have to do is make sure the component model is implemented by the host, i.e. the Miden VM.

@bobbinth I'm interested in what you think about that as well. Since we have some plans to rework the way memory is implemented on the roadmap anyway, perhaps one of the ways we end up refactoring it is to have a model which more closely matches that of WebAssembly and the component model. The way it works in the component model is that within a component, memory is shared, but between components, memory must be shared explicitly. Resources are particularly interesting, because the resource itself can reference memory in its context, so external users of the resource can call its APIs and they "just work"; however even if those external users could see the internal representation of the resource, they can't access the addresses (because they are meaningless unless the actual memory has been shared with you).

We're not running Wasm modules of course, and things are slightly different with call (namely the semantics are more like dynamically instantiating a new component instance on each call, which is technically a supported use case with the component model). The primary difficulty in adapting it to Miden IMO, is that the VM has to be aware of components, or we lose some of the best aspects of the design. Compiling a set of components to Miden today would result in them all executing in the same memory (which would not be correct). Even if we assume that each component got its own memory, compiling a component that exports a resource would not work properly across a call boundary, because if the callee call's one of the resource's methods, that code would execute in the callee's context, not the context the resource was created in. Probably the only place where it would work today without any changes is resources defined in a kernel module (if we assume the resource methods are invoked with syscall), but that's of limited utility, and accidental at best.

Realistically, I think that translates into the following changes:

Require describing MASM code in terms of components
Make the VM aware of component packages, so that it can load, link, and execute code in terms of components
Internally in the VM, we'd need to rework the current "context" system to use a representation more akin to component instances, which are basically the contexts we're familiar with, but are instantiated in terms of components, rather than ad-hoc, and contain additional auxiliary data needed for some of the component model plumbing
Cross-component calls would need to be intercepted by the VM to do a context switch, which is facilitated by the component model itself
When executing a call, we'd want to dynamically create a new instance of the component containing the callee, and then handle it as a regular context switch
With the previous two items, it's not clear we'd need to treat kernel modules as a special thing anymore (or even have the syscall instruction), since kernel modules are effectively a share-nothing component.

This is just an ad-hoc sketch of what I think that would look like, but I think it would be a worthwhile exercise to work through this and see how it would actually look implementing it in Miden VM, and what problems we have on our roadmap that might get solved/simplified by doing so.

Bobbin Threadbare · Answer 10 · Wed Dec 06 2023 15:00:01 GMT+0800 (China Standard Time)

One issue which I'm not yet able to solve with making components first-class citizens within the VM is how to identify components. So, I'm curious how this is handled in WASM (or other places).

For example, let's say I have a component for computing SHA256 hashes. This component has a starter procedure (which initializes it's memory) and has a couple of public procedures - e.g., hash1to1 and hash2to1.

Assuming we create a new memory context per component, we need some way to associate a given component with its memory context. Currently, memory contexts in Miden VM have unique identifiers (we can think about this identifier as a 32-bit value). So, somehow we need to be able to map all calls to sha256::hash1to1 and sha256::hash2to1 to a unique ID. And I believe this mapping needs to be done at runtime (though, not 100% sure about this).

Denys Zadorozhnyi · Answer 11 · Fri Dec 08 2023 00:47:43 GMT+0800 (China Standard Time)

One issue which I'm not yet able to solve with making components first-class citizens within the VM is how to identify components. So, I'm curious how this is handled in WASM (or other places).

Every component in Wasm should have a package in the form of namespace:name@version (version is optional).

Bobbin Threadbare · Answer 12 · Fri Dec 08 2023 06:57:16 GMT+0800 (China Standard Time)

Right now, every memory context has a unique ID which is a 32-bit integer. We can pretty easily extended to a full field element (i.e., almost 64 bits) but making the identifier bigger (e.g., a full word) would come with noticeable performance penalties.

But putting the size of the identifier aside, the question is how do we specify which context we want to enter. Right now, call instruction works like this:

When we say call.some_module::some_procedure, this gets actually translated into something like CALL(0x1234, clk), where 0x1234 is the MAST root of the procedure to be called and clk is the current clock cycle used as the unique identifier for the new context.

One thing to point out: at the VM level there is no notion of packages, modules, or namespaces. So, it doesn't really matter where some_procedure is located. As long as its MAST root is 0x1234 it could be anywhere and the VM would not know the difference.

We could modify the semantics of the call operation to work like so: CALL(mast_root, stack_top) where stack_top is the element on the top of the stack. This way, we can specify which procedure to invoke and which context to invoke it in. (one interesting thing here is that syscall becomes just CALL(mast_root, 0) because for syscalls the target context is always $0$).

This could work very well for something like accounts, where we could use account ID as the context ID. But it is not clear to me what we could use for things like notes. If we could use words for context identifiers, then we could use note hash, but as mentioned above, this will likely have non-negligible performance implications.

The other problem is how to enforce that procedure with MAST root 0x1234 can be called in a given context. Right now, we impose no such restrictions at the VM level (any procedure can be executed in any context, with syscalls being the only exception). The methodology we use for enforcing access control for syscalls is not scalable - i.e., we can't have more than a handful of contexts controlled in such a way.

So, basically there are two open questions:

Is there an efficient way to use words for context identifiers?
How can we efficiently implement "access control" for each context at the VM level?

If we can solve these, I think we can support a fully-functional component model inside the VM.

RFC: Miden SDK Outline

Footnotes

Expressing Accounts, Notes, and Transaction Scripts

Footnotes