lightninglabs / falafel

Go tool to generate go APIs for gRPC services for use on mobile/WASM platforms.

falafel

falafel is a protoc plugin written in go that is used to generate gomobile compatible APIs for gRPC services for use on mobile platforms.

Currently being used with lnd.

Description

falafel translates protobuf definitions to gomobile compatible APIs. Behind this API we directly talk to the gRPC server using an in-memory gRPC client, ensuring all communication happens in-process using serialized protocol buffers, without needing to expose the gRPC server on an open port. To support streaming RPCs, like subscribing to real-time updates, callbacks are provided for all APIs.

The gRPC server must support using custom listeners.

Getting started

Pass the falafel plugin to protoc with custom options.

Here is an example how falafel is used with lnd:

falafel=$(which falafel)

# Name of the package for the generated APIs.
pkg="lndmobile"

# The package where the protobuf definitions originally are found.
target_pkg="github.com/lightningnetwork/lnd/lnrpc"

# A mapping from grpc service to name of the custom listeners. The grpc server
# must be configured to listen on these.
listeners="lightning=lightningLis walletunlocker=walletUnlockerLis"

# Set to 1 to create boiler plate grpc client code and listeners. If more than
# one proto file is being parsed, it should only be done once.
mem_rpc=1

opts="package_name=$pkg,target_package=$target_pkg,listeners=$listeners,mem_rpc=$mem_rpc"
protoc -I/usr/local/include -I. \
       -I$GOPATH/src/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis \
       --plugin=protoc-gen-custom=$falafel\
       --custom_out=./build \
       --custom_opt="$opts" \
       --proto_path=../lnrpc \
       rpc.proto

With the go bindings generated, define an entry point for the application to start the gRPC service:

func Start() {
	// We call the main method with the custom in-memory listeners called
	// by the mobile APIs, such that the grpc server will use these.
	cfg := lnd.ListenerCfg{
		WalletUnlocker: walletUnlockerLis,
		RPCListener:    lightningLis,
	}

	go func() {
		if err := lnd.Main(cfg); err != nil {
			if e, ok := err.(*flags.Error); ok &&
				e.Type == flags.ErrHelp {
			} else {
				fmt.Fprintln(os.Stderr, err)
			}
			os.Exit(1)
		}
	}()
}

The gRPC server should be started by listening on the passed listeners.

Compiling with gomobile

Package lndmobile is now ready to be cross-compiled using gomobile:

gomobile bind -target=ios github.com/lightningnetwork/lnd/mobile

Generating JSON/WASM stubs

falafel was initially built as a code generator specifically for generating gomobile compatible RPC stubs for lnd. But because generating stub code from protobuf files is a very useful task, falafel also has a secondary operating mode in which it generates stubs for interacting with a gRPC interface from a JSON/WASM context.

What are JSON/WASM stubs?

In short, the JSON stubs generated by falafel is helper code that allows a dynamic language environment that uses JSON as its main data structure (e.g. JavaScript code running in a browser) to interact with a gRPC client that is running in the same browser but for example in a WASM context.

Or in other words: The stubs convert a JSON request into a proper gRPC request, send it to the gRPC server and translate the response back into JSON.

The stubs could also be described as doing the reverse of what grpc-gateway does, on the client side, translating between JSON and native gRPC.

Example of generated code

For our main example, we assume the following stateservice.proto file:

syntax = "proto3";
package lnrpc;
option go_package = "github.com/lightningnetwork/lnd/lnrpc";

service State {
  rpc SubscribeState (SubscribeStateRequest)
      returns (stream SubscribeStateResponse);
  rpc GetState (GetStateRequest) returns (GetStateResponse);
}

enum WalletState {
  NON_EXISTING = 0;
  LOCKED = 1;
  UNLOCKED = 2;
  RPC_ACTIVE = 3;

  WAITING_TO_START = 255;
}
message SubscribeStateRequest {
}
message SubscribeStateResponse {
  WalletState state = 1;
}
message GetStateRequest {
}
message GetStateResponse {
  WalletState state = 1;
}

Running falafel with:

FALAFEL_BIN=$(which falafel)
opts="package_name=lnrpc,js_stubs=1,build_tags=// +build js"
protoc -I/usr/local/include -I. -I.. \
    --plugin=protoc-gen-custom=$FALAFEL_BIN\
    --custom_out=. \
    --custom_opt="$opts" \
    stateservice.proto

will then generate the following stub file:

// Code generated by falafel 0.9.1. DO NOT EDIT.
// source: stateservice.proto

// +build js

package main

import (
	"context"

	gateway "github.com/grpc-ecosystem/grpc-gateway/v2/runtime"
	"github.com/lightningnetwork/lnd/lnrpc"
	"google.golang.org/grpc"
	"google.golang.org/protobuf/encoding/protojson"
)

func RegisterStateJSONCallbacks(registry map[string]func(ctx context.Context,
	conn *grpc.ClientConn, reqJSON string, callback func(string, error))) {

	marshaler := &gateway.JSONPb{
	    MarshalOptions: protojson.MarshalOptions{
		    UseProtoNames:   true,
		    EmitUnpopulated: true,
	    },
	}

	registry["lnrpc.State.SubscribeState"] = func(ctx context.Context,
		conn *grpc.ClientConn, reqJSON string, callback func(string, error)) {

		req := &lnrpc.SubscribeStateRequest{}
		err := marshaler.Unmarshal([]byte(reqJSON), req)
		if err != nil {
			callback("", err)
			return
		}

		client := lnrpc.NewStateClient(conn)
		stream, err := client.SubscribeState(ctx, req)
		if err != nil {
			callback("", err)
			return
		}

		go func() {
			for {
				select {
				case <-stream.Context().Done():
					callback("", stream.Context().Err())
					return
				default:
				}

				resp, err := stream.Recv()
				if err != nil {
					callback("", err)
					return
				}

				respBytes, err := marshaler.Marshal(resp)
				if err != nil {
					callback("", err)
					return
				}
				callback(string(respBytes), nil)
			}
		}()
	}

	registry["lnrpc.State.GetState"] = func(ctx context.Context,
		conn *grpc.ClientConn, reqJSON string, callback func(string, error)) {

		req := &lnrpc.GetStateRequest{}
		err := marshaler.Unmarshal([]byte(reqJSON), req)
		if err != nil {
			callback("", err)
			return
		}

		client := lnrpc.NewStateClient(conn)
		resp, err := client.GetState(ctx, req)
		if err != nil {
			callback("", err)
			return
		}

		respBytes, err := marshaler.Marshal(resp)
		if err != nil {
			callback("", err)
			return
		}
		callback(string(respBytes), nil)
	}
}

An example WASM client can then be built to bridge the gap between JavaScript and the native gRPC client.

// +build js

package main

import (
	"context"
	"runtime/debug"
	"syscall/js"

	"google.golang.org/grpc"
	
	// Import the generated JSON stubs from the lnrpc package where we generated
	// them before
	_ "github.com/lightningnetwork/lnd/lnrpc"
)

var (
	lndConn *grpc.ClientConn

	registry = make(map[string]func(ctx context.Context,
		conn *grpc.ClientConn, reqJSON string,
		callback func(string, error)))
)

func main() {
	defer func() {
		if r := recover(); r != nil {
			log.Debugf("Recovered in f: %v", r)
			debug.PrintStack()
		}
	}()

	// Setup JS callbacks.
	js.Global().Set("wasmClientInvokeRPC", js.FuncOf(wasmClientInvokeRPC))
	lnrpc.RegisterStateJSONCallbacks(registry)

	// Setup native gRPC connection to lnd, the stubs will translate calls to
	// this connection.
	lndConn = connectToLnd()

	// Wait for interrupt signal here or do other stuff...
}

func wasmClientInvokeRPC(_ js.Value, args []js.Value) interface{} {
	if len(args) != 3 {
		return js.ValueOf("invalid use of wasmClientInvokeRPC, " +
			"need 3 parameters: rpcName, request, callback")
	}

	if lndConn == nil {
		return js.ValueOf("RPC connection not ready")
	}

	rpcName := args[0].String()
	requestJSON := args[1].String()
	jsCallback := args[len(args)-1:][0]

	method, ok := registry[rpcName]
	if !ok {
		return js.ValueOf("rpc with name " + rpcName + " not found")
	}

	go func() {
		log.Infof("Calling '%s' on RPC with request %s",
			rpcName, requestJSON)
		cb := func(resultJSON string, err error) {
			if err != nil {
				jsCallback.Invoke(js.ValueOf(err.Error()))
			} else {
				jsCallback.Invoke(js.ValueOf(resultJSON))
			}
		}
		ctx := context.Background()
		method(ctx, lndConn, requestJSON, cb)
		<-ctx.Done()
	}()
	return nil
}

A website can then call into those functions with a little bit of JavaScript (we assume here that the WASM binary was already loaded and initialized correctly for this example):

<html>
<body>

<button onClick="callWASM('GetState', '{}');">GetState</button>

<script>
    async function callWASM(rpcName, req) {
        wasmClientInvokeRPC('lnrpc.Lightning.'+rpcName, req, setResult);
    }
    
    function setResult(result) {
        console.log("Got result from RPC: " + JSON.stringify(result));
    }
</script>
</body>
</html>