bufbuild / protobuf-es

Sorry if this is a noob question, I'm quite new to gRPC in general.

I'm in the process of writing a protoplugin for grpc-gateway client SDK, I decided to use the Protobuf-ES framework and it works really great, thanks for that 🙇

I encountered a problem, where I want to support protoc-gen-openapiv2 family of options, e.g. the message option in

syntax = "proto3";

import "protoc-gen-openapiv2/options/annotations.proto";

message OptionMessage {
  option (grpc.gateway.protoc_gen_openapiv2.options.openapiv2_schema) = {
    json_schema: {
      required: [
        "foo"
      ]
    }
  };
  string foo = 1;
};

When I convert this example with buf build --output -#format=json, with all the appropriate dependencies, I obtain a JSON representation of file descriptor that contains an options field populated with [grpc.gateway.protoc_gen_openapiv2.options.openapiv2_schema] option. When I pass this JSON to the FileDescriptorSet from @bufbuild/protobuf the instance of FileDescriptorSet then contains no options at all, therefor my plugin is unable to process the option. I stepped through the code and ended in

protobuf-es/packages/protobuf/src/private/json-format-common.ts

Line 66 in c89f712

export function makeJsonFormatCommon(

Which is a little bit complicated to follow.

I was thinking that I will be able to extract the option with findCustomMessageOption API but when the option is missing in the FileDescriptorSet instance it is helpless.

I created a repro through a test in Codesandbox. To run the test, just use the integrated terminal and run yarn test.

Thank you for the detailed issue, Václav!

The issue here is that this implementation don't support extensions yet (tracked in #86).

The option you are specifying is technically an extension of the message google.protobuf.MessageOptions, and is serialized to JSON like a regular field, but with the name in brackets. When parsed again with FileDescriptorSet.fromJson, we don't recognize that it's an extension, and the option value is not available.

However, extracting the custom option should work if you simply store the set of descriptors in binary format instead of JSON. In the binary format, the value set by the custom option is retained as an unknown field.

Let me know if that helps. We plan to support extensions soon.

extracting the custom option should work if you simply store the set of descriptors in binary format

That helped, thank you 🙇

@czabaj, support for extensions just landed, and will be part of the next release.

Here is a guide on how to use an extension to read a custom option: writing_plugins.md. Exactly the same pattern will work for the option grpc.gateway.protoc_gen_openapiv2.options.openapiv2_schema.

First you generate code for the annotations with buf generate buf.build/grpc-ecosystem/grpc-gateway. This generates the extension openapiv2_swagger in annotations_pb.ts. In your plugin, you can get the option from the message descriptor's options using getExtension and the extension.

For your custom plugin, this would be the recommended approach. Just as a side-note: With the new support for extensions, you can now also parse and serialize the output from buf build in JSON format, retaining custom options. The following function takes a binary file descriptor set as input (buf build -o), creates a registry for all described types, and uses the extensions from this registry to serialize the custom options to JSON:

import {readFileSync} from "node:fs";
import {createRegistryFromDescriptors, FileDescriptorSet} from "@bufbuild/protobuf";

function binaryFileDescriptorSetToJson(bytes: Uint8Array): string {
  const set = FileDescriptorSet.fromBinary(bytes);
  const typeRegistry = createRegistryFromDescriptors(set);
  return set.toJsonString({
    typeRegistry,
  });
}

The `FileDescriptorSet` instance omits my option