Ability to generate source relative files (`paths=source_relative`)

[Leandro Branco]

It would be nice and useful to be able to generate files in a relative path (paths=source_relative) to the source .proto definitions like go, go-grpc and connect-go do:

  - plugin: buf.build/bufbuild/connect-go:v1.10.0
    out: gen
    opt:
      - paths=source_relative

[Timo Stamm]

Leandro, could you explain a bit what you expect?

The option is documented with:

If the paths=source_relative flag is specified, the output file is placed in the same relative directory as the input file. For example, an input file protos/buzz.proto results in an output file at protos/buzz.pb.go.

This is the behavior with protoc-gen-es out of the box. The input file protos/buzz.proto results in protos/buzz_pb.js.

[Leandro Branco]

Hi @timostamm, thanks for the prompt response.

I've just created a small repo to show the case: https://github.com/leandro-branco/protobuf-es-example.
When I run, buf generate, the package/acme/example/protos/api/v1/hello.proto is created at example/protos/api/v1/hello_pb.ts in the root folder (and not in package/acme/example/protos/api/v1/hello_pb.ts as expected).

Creating the repo, I've realised it seems to be something related to buf workspaces. When trying to define buf workspace with multi module, for instance, a module in example/acme folder, buf doesn't generate in the correct folder.

Without buf.work.yaml, it works.

I'm also not able to config the buf.gen.yaml to output files in a relative folder to the proto source like:

version: v1
plugins:
  - plugin: buf.build/bufbuild/es:v1.3.0
    out: ./gen
    opt:
      - target=ts

Files are generated at gen folder in the root whereas I was expecting to be generated at package/acme/example/protos/api/v1/gen/hello_pb.ts.

Edit:
Running another test, if out: .. when generating, files are created in the parent folder:
../package/acme/example/protos/api/v1/hello_pb.ts (relative to the root folder)

Edit 2:
When setup out: ., I'd say it only works because protobuf-es is creating files using the path ./package/acme/example/protos/api/v1/hello_pb.ts which coincides with the proto definition path.

It seems the protobuf-es is appending the fullname path from here with the out definition.

Let me know if you need any help to reproduce it please. Thank you!

[Timo Stamm]

With this buf.work.yaml the buf CLI will use packages/acme as the module root. You can see the files in this module with:

$ cd packages/acme 
$ buf ls-files
example/protos/api/v1/hello.proto

So the input file is example/protos/api/v1/hello.proto, and our plugin will generate a corresponding example/protos/api/v1/hello_pb.ts.

If you run buf generate from the repository root, the buf CLI writes the file to example/protos/api/v1/hello_pb.ts, relative to the repository root and the plugin out parameter (.). The plugin is unaware where the source files are on disk, and where the generated files are written to.

[Leandro Branco]

Thank you for the explanation @timostamm!

Another related question is that even without buf.work.yaml, the plugin is generating files relative to the repository root when the plugin out parameter is ..

That was the motivation to open the issue in the first place. It'd be nice to be able to generate relative to source files, instead. For example, setting the out parameter as ./gen and having the files generated to [root]/package/acme/example/protos/api/v1/gen folder and not to the [root]/gen.

[Timo Stamm]

I agree that this would be nice - the user experience with buf workspaces is clearly not ideal here - but the plugin really is unaware where the source files are on disk. In the CodeGeneratorRequest it receives, the input file has the path example/protos/api/v1/hello.proto, and in the CodeGeneratorResponse it produces, the output file has the path example/protos/api/v1/hello_pb.ts.

The only way to achieve that output files are put to packages/acme would be to have a plugin option where you can specify output_path_prefix=packages/acme. And this is something that can already be done with the out option in buf.gen.yaml, or or the --es_out flag in protoc.

I recommend a very different approach here: Remove buf.work.yaml and create a proto directory at the root of your repository:

$ tree proto
proto
├── acme
│   └── example
│       └── api
│           └── v1
│               └── hello.proto
└── buf.yaml

In your buf.gen.yaml, set out: packages/acme/example/src/gen.

Now you can lint with buf lint proto and generate with buf generate proto. Note that this also fixes a lint issue with your setup - we recommend that the protobuf package structure matches the directory structure of the module.

[Leandro Branco]

That's great. I'll adapt the solution you gave to my actual repo but, once again, thank you very much!
Very insightful solution and how the plugin works.