Conformance tests
noahseger opened this issue · comments
Background: https://buf.build set up a conformance testing framework for various JS runtimes to test and prove their own protobuf-es runtime implementation. I contributed a patched protoscript test there ( bufbuild/protobuf-conformance#24.
We are deploying TwirpScript increasingly on the server side, and it's been a great simple RPC framework. I'm only interested in avoiding as many serialization headaches as possible :)
I'm opening this issue in TwirpScript mostly to start a discussion about whether it:
A) Is worth understanding and closing the compliance gaps
B) Makes sense to converge on an existing compliant runtime
It's possible that protobuf-es doesn't meet the project goals of TwirpScript. For example, something that seems minor but may be a big difference is their representation of nested messages as optional, regardless of the optional keyword being present in the definition. They also have a larger runtime, and it's not clear how much of it is tree-shakeable.
Hey @noahseger. First off, this is awesome. I've wanted to run Google's protobuf conformance tests against protoscript for awhile.
I'm happy to consider both A and B. Originally, when I "wrote" protoscript there weren't any strong protobuf contenders that I was aware of to support browser clients.
Bandwidth-wise, I'm happy to close the compliance gap. That said, as a user given both options: protoscript with 100% compliance or TwirpScript using protobuf-es under the hood, which would you prefer?
@tatethurston I think if I had to say now, I'd choose converging on protobuf-es IFF the runtime philosophies align closely enough. But you're also making me think that being able to "hook" TwirpScript with your runtime could also be extremely useful.
For example, we already had a request to trade off any protobuf runtime at all and consume only literal application/json. But without a super lightweight runtime it's a bit dangerous (types lie, no defaults, etc). If I could plug a JSON-only runtime it could be typesafe, and only dozens/hundreds of bytes per service.
Thanks, and happy to explore and contribute in whatever direction ⚡
Thanks @noahseger
For example, we already had a request to trade off any protobuf runtime at all and consume only literal application/json. But without a super lightweight runtime it's a bit dangerous (types lie, no defaults, etc). If I could plug a JSON-only runtime it could be typesafe, and only dozens/hundreds of bytes per service.
Where does TwirpScript / protoscript fall short on this for you today? The protobuf runtime should be dead code eliminated from any builds that only use JSON.
Hmm, we were not seeing treeshaking work with esbuild or webpack, I think because the JSON request/response objects call the protobuf initialize() method, and I thought it was a fundamental limitation.
But after working a bit with the internals for the test runner, maybe it's just a bug? Are these lines supposed to reference ${tsTypeJSON}.initialize() instead, and that's what's pulling the runtime in? https://github.com/tatethurston/ProtoScript/blob/main/src/codegen/autogenerate/index.ts#L599-L604
Ah yes that looks like a bug -- there are Message and MessageJSON entry points to separate the protobuf and JSON run times. initialize calls from JSON methods should reference the MessageJSON.initialize calls eg: https://github.com/tatethurston/TwirpScript/blob/main/examples/typescript-fullstack/src/protos/haberdasher.pb.ts#L194
Implemented in ProtoScript, in package e2e/conformance
Related: tatethurston/ProtoScript#29 tatethurston/ProtoScript#31 tatethurston/ProtoScript#32.