bufbuild / protobuf-es

It's currently not possible to load a .proto file with protobufjs and turn it into a descriptor with @bufbuild/protobuf.

So far, debugging lead me to this:

protobufjs does not resolve field type names for message type references to their fully qualified names.
@bufbuild/protobuf however expects those to be fully resolved in newFields

protobuf-es/packages/protobuf/src/create-descriptor-set.ts

Line 537 in ae29a60

const message = cart.messages.get(trimLeadingDot(proto.typeName));

This leads to error: invalid FieldDescriptorProto: type_name <TYPE NAME> not found

Even adding the fully qualified names in the source .proto doesn't resolve it which leads me to believe that protobufjs actually strips those (expecting them to be resolved by parent lookup in the consumer?)

The "JSON descriptors" protobuf.js uses and produces are conceptually similar to the google/protobuf/descriptor.proto messages, but they are not the same. Unfortunately, they only contain a subset of the information in descriptor.proto, and the format is unique to protobuf.js.

protobufjs/ext/descriptor is an experimental extension to convert the protobuf.js descriptors to protobuf messages.

Are you aware of any alternatives by chance? I'm trying to load raw .proto files at runtime and use them with @bufbuild/proto if at all possibe.

EDIT: Note that I'm currently using .toDescriptor from the extension you linked. I'm not using the JSON descriptors. So the problem I'm describing here is happening with the Descriptor.IDescriptorProto from /ext/descriptor.

I fear I might_ have to resort to exec with buf build to produce the file descriptor set :-/

I'm not aware of a really awesome solution. I would try to avoid going from protobuf.js descriptors to descriptor.proto to @bufbuild/protobuf messages. Because protobuf.js descriptors are a subset of descriptor.proto, some data needs to be synthesized to produce descriptor.proto, and this is likely to cause problems. I have a hunch that relative references in type names are just the tip of the ice berg.

@bufbuild/protobuf messages only need a subset of the information in descriptor.proto. protobuf.js descriptor might have all necessary data to go from them to @bufbuild/protobuf directly. But I would take a closer look first, especially regarding syntax. (@bufbuild/protobuf makes a distinction between proto2 and proto3 that protobuf.js might just not care about.)

Shelling out to the buf CLI is definitely the most compatible approach, and also the easiest and most performant, if you can run the binary. It works for esbuild too 🙂 With spawn() from node:child_process, you can pipe to descriptors straight into memory.

Thanks. I implemented it using spawn for now. It's not ideal because of the restrictions that puts on portability but it works ;-).

Ty!

That was fast! Did you know we have a rudimentary starting point for a JS parser here? 😇

Omg, I love y'all. <3 That's great! I'll check that out next week after the Diablo IV release :-P

Incompatibility between `protobufjs` and `@bufbuild/protobuf`