how to use buf in an offline env?
dobet opened this issue · comments
if I use buf to build protobuf in an offline env. can't online download deps.
|-- Makefile
|-- api
| `-- v1
| `-- api_service1
| `-- api.proto
|-- rpc
| |-- Makefile
| |-- build_conf.py
| `-- rpc_service1
| `-- rpc1.proto
`-- third_party
|-- buf.yaml
|-- errors
| `-- errors.proto
|-- gnostic
| `-- openapi
| |-- discovery
| | `-- v1
| | `-- discovery.proto
| |-- v2
| | `-- openapiv2.proto
| `-- v3
| |-- README.md
| |-- annotations.proto
| `-- openapiv3.proto
`-- google
|-- api
| |-- annotations.proto
| |-- field_behavior.proto
| |-- http.proto
| `-- httpbody.proto
`-- protobuf
|-- any.proto
|-- api.proto
|-- descriptor.proto
|-- duration.proto
|-- empty.proto
|-- field_mask.proto
|-- source_context.proto
|-- struct.proto
|-- timestamp.proto
|-- type.proto
`-- wrappers.proto
if use protoc, I can use "--proto_path=third_party", in buf, what is this function?
Hey! You can use workspaces.
I'm not 100% sure, but I think you're misunderstanding what he's asking. I think he's asking how to use buf generate offline. I ran into the same issue where I was on a plane doing some coding, and discovered that there's no way to run buf offline:
🐚 ~/dev/ftl $ (cd protos && buf generate)
Failure: the server hosted at that remote is unavailable. Are you sure "buf.build" is a valid remote address?
It would be great if buf generate had an --offline flag which reused cached modules from the previous run, or even better if it just gave a warning and continued.
The need to contact a server every time I want to generate also seems to make buf run quite slowly, taking 2+ seconds just to regenerate a couple of files:
🐚 ~/dev/ftl $ time (cd protos && buf generate)
( cd protos && buf generate; ) 0.31s user 0.07s system 17% cpu 2.181 total
buf does have a module cache (it's at ~/.buf/cache if you're interested in checking it out), so unless you're using remote plugins, you should have no problem when offline. If you can provide us with specific reproduction steps demonstrating buf not working with previously-downloaded modules, we'd be happy to investigate.
That directory doesn't exist on my system (Mac), nor is it in ~/Library/Caches:
🐚 ~/…/ftl/protos $ ls -d ~/.buf ~/Library/Caches/buf
ls: /Users/alec/.buf: No such file or directory
ls: /Users/alec/Library/Caches/buf: No such file or directory
Configuration below. I do use remote plugins (though I can't tell if this error message is triggered by plugins or modules), but what I'm suggesting is that either way, if I have local copies of the plugins or modules (which I should), then the buf tool shouldn't fail, only warn:
🐚 ~/…/ftl/protos $ cat buf.yaml
version: v1
breaking:
use:
- FILE
lint:
use:
- BASIC
ignore_only:
FIELD_LOWER_SNAKE_CASE:
- xyz/block/ftl/v1
🐚 ~/…/ftl/protos $ cat buf.gen.yaml
version: v1
plugins:
- plugin: go
out: .
opt: paths=source_relative
- plugin: connect-go
out: .
opt: paths=source_relative
- plugin: buf.build/bufbuild/es:v1.4.0
out: ../frontend/src/protos
opt: target=ts
- plugin: buf.build/connectrpc/es:v1.1.2
out: ../frontend/src/protos
opt: target=ts
🐚 ~/…/ftl/protos $ buf generate
Failure: the server hosted at that remote is unavailable. Are you sure "buf.build" is a valid remote address?
I did find ~/.cache/buf but it appears to be empty:
🐚 ~/…/ftl/protos $ find ~/.cache/buf/v2
/Users/alec/.cache/buf/v2
/Users/alec/.cache/buf/v2/module
🐚 ~/…/ftl/protos $ find ~/.cache/buf/v1
/Users/alec/.cache/buf/v1
/Users/alec/.cache/buf/v1/module
/Users/alec/.cache/buf/v1/module/lock
/Users/alec/.cache/buf/v1/module/sum
/Users/alec/.cache/buf/v1/module/data
Plugins are remote by design, unfortunately in our opinion. Because of the myriad ways that plugins are developed and executed (native binaries compiled from various languages, JARs, npm), the lowest-common-denominator way to run all of them is by packaging them up into a Docker container. We didn't want to make buf take on Docker as a dependency that you need to have available to run remote plugins, so for now, you can only use remote plugins when connected to your BSR instance (buf.build in this case). We've undertaken early work to create a WASM-based plugin ecosystem for Protobuf, and buried in the buf code is support for WASM plugins, hidden behind an environment variable. However, this really requires a lot of buy-in across the Protobuf ecosystem that we haven't marshalled just yet, and WASM is still pretty finicky across all the languages users write Protobuf plugins in. So candidly, this effort has stalled.
You don't have any module dependencies in your buf.yaml, so you don't have anything in your module cache - that part makes sense. The connection error is solely from the usage of remote plugins. Commands like buf build, buf lint, buf breaking etc will always work offline, including if you have a dependency on a BSR module (as it will be cached), but buf generate with remote plugins will not unfortunately.
We'd love to invest more here, and we think the right way is via WASM (after investigating a lot of avenues), but it just hasn't been our highest priority item as of yet.
Ah okay, that makes total sense. I had a misunderstanding of how remote plugins worked TBH, I assumed they were hosted remotely but synced down locally, but it sounds like the execution of the plugin is actually hosted remotely and the client sends the descriptor set to the server for execution. This makes much more sense than my misconception ;)
Thanks for the explanation, and I agree that with an ecosystem like protobuf's, WASM would be a tough task.
No worries, and sorry for both the trouble and the delay in responding, it's been a hectic week.
Closing this issue for now. Let us know if you need anything else.
@bufdev sorry for asking in a closed issue but any condition required for using buf lint in offline env?
I can run buf lint on my laptop without internet but when I run the step on our restricted network on CI then I see the issue > Failure: the server hosted at that remote is unavailable.
ah ok, I found the buf build mod cache folder in the comments already.