feat(ts): protobuf-es runtime mode for TS client/server (native messages + sebuf transport)

[hishamank]

What

Adds an opt-in ts_runtime=protobuf-es mode to protoc-gen-ts-client and protoc-gen-ts-server. In this mode, message types + JSON codec come from Buf's native protoc-gen-es (*_pb.ts), and sebuf generates only the HTTP transport on top — routing every request/response through create/toJson/fromJson(..., { ignoreUnknownFields: true }). This mirrors how Go uses protoc-gen-go + protoc-gen-go-http.

The existing hand-rolled mode stays the default — no behavior change unless ts_runtime=protobuf-es is passed.

Why

sebuf's Go output uses the language's native protobuf runtime, so it can't drift from protojson. The TS output hand-rolls message types with no codec, so it can silently disagree with the wire — e.g. a server omits zero-values per protojson, but the hand-rolled TS types declare them present, so resp.list.map(...) throws on an empty list. This brings TS into the native-runtime tier. It is not adopting Connect — sebuf keeps its own HTTP transport (paths, headers, ValidationError); only the message layer goes native.

How it works

• New ts_runtime=protobuf-es plugin option (invalid values fail loudly).
• es-mode skips emitting the hand-rolled type modules (protoc-gen-es owns them) but still emits the shared errors.ts.
• Messages import <Msg>Schema (value) + <Msg> (type) from ./<proto>_pb.js.
• The @bufbuild/protobuf import lists only the symbols actually used: each es-mode emission site records the helpers it prints (MessageInitShape/fromJson per method, create/toJson only when a body or response is encoded), so the import mirrors real usage and compiles under noUnusedLocals.
• Client: MessageInitShape<typeof ReqSchema> params; body: JSON.stringify(toJson(ReqSchema, create(ReqSchema, req))); fromJson(ResSchema, await resp.json(), { ignoreUnknownFields: true }).
• Server: fromJson request decode; toJson(create(ResSchema, result)) response encode; handler takes branded <Req>, returns MessageInitShape<...>.
• Unary and server-streaming (SSE) supported.

Review fixes (second round of commits)

A review pass found two real issues in the first iteration; both are fixed on this branch:

1. Latent hand-rolled @bufbuild/protobuf import (fixed in 8c1e4c1). The runtime-symbol import list was originally derived by regex-scanning the generated body, and the scan ran in both modes. An RPC literally named Create renders async create(, which matched \bcreate\b and injected import { create } from "@bufbuild/protobuf" into hand-rolled output — a package hand-rolled consumers don't install. A field named create in an es-mode GET method had the mirror problem: an unused import that fails noUnusedLocals. The body scan is now deleted; each es-mode emission site records exactly the helpers it prints, and NeedProtobufES is a no-op outside es mode as a structural invariant. Regression tests cover both failure modes (runtime_symbols_test.go in both generators), and all pre-existing goldens are byte-identical under the new tracking.
2. es output was never machine-typechecked (fixed in 96b33c9). The es goldens' typecheck was a manual verification, and it turned out the checked-in golden tree didn't actually compile: the fixtures' *_pb.ts import ./sebuf/http/annotations_pb.js, which was never checked in (TS2307). New TestTSClientGenESGoldenTypecheck / TestTSServerGenESGoldenTypecheck run tsc --noEmit (strict nodenext + noUnusedLocals, via the ported tscommon/typecheck harness) over testdata/golden/es against the real @bufbuild/protobuf package, skipping cleanly when the toolchain is absent. The missing annotations_pb.ts goldens are now checked in. This is also the machine guard for ESQualifiedName drift: a symbol-name divergence from protoc-gen-es now fails CI with a TS2724: no exported member error instead of surfacing in a consumer build (verified by fault injection).

Also documented (5854efd): URL path/query params do not route through the codec (only bodies and SSE events do — by design, URL params have no protojson form), and MessageInitShape makes all request fields optional, so a missing path param compiles and only surfaces as a runtime "undefined" URL segment.

Tests / verification

• go test ./..., gofmt, vet, staticcheck all green.
• Hand-rolled default mode is byte-identical (all pre-existing goldens unchanged; new golden coverage lives under testdata/golden/es/).
• es goldens are typechecked in CI (tsc --noEmit, strict nodenext, noUnusedLocals) against the real @bufbuild/protobuf package — no longer a manual claim.
• Regression tests assert hand-rolled output never imports @bufbuild/protobuf (RPC named Create) and that es-mode imports only the runtime symbols actually used (GET method with a field named create).
• Wire-conformance test proves: omitted zero-values are materialized on decode (scalars → 0/""/false, lists → []), and unknown fields don't throw (ignoreUnknownFields) — but do throw without the flag.
• Golden es tests cover unary POST, GET, and server-streaming (SSE); a non-enum GET fixture and enum-param rejection are asserted.

Known limitations (documented in docs/client-generation.md)

• Enum path/query parameters are not yet supported in es-mode (protobuf-es enums are numeric; string↔enum conversion isn't implemented). The generator fails loudly at generation time rather than emitting uncompilable code.
• RPC I/O should be top-level messages (nested names reconciled via ESQualifiedName, verified against @bufbuild/protoc-gen-es@2.12.1; the es golden typecheck catches drift on other versions).
• URL parameters bypass the codec (string-coerced from the init shape) and path params are not compile-time required under MessageInitShape — see docs.

Base

Targets feat/ts-modules-oneof-integration (the merged #199 + #200 result) since the es transport reuses that modules-layout import machinery. Rebase onto main once those land.

🤖 Generated with Claude Code

[SebastienMelki]

The approach is right; the wire contract is the problem.

Using the official protoc-gen-es types and building sebuf's transport around them is exactly what we do in Go, and it's the correct direction. But that's precisely the bar this has to clear: in Go we don't hand the raw runtime encoder the wire — we wrap protoc-gen-go with a sebuf marshal layer (MarshalJSONSebuf, dispatched at internal/httpgen/generator.go:808) that renders sebuf's JSON, which deliberately diverges from canonical protojson wherever a sebuf.http annotation is set.

This PR uses protoc-gen-es types and its raw canonical codec (toJson/fromJson) — it skips the equivalent of that marshal layer. So for any annotated proto, es mode is on a different wire than both the sebuf Go server and our current hand-rolled TS client. It is not a drop-in replacement; it's a second, incompatible wire format that happens to coincide only when no JSON-mapping annotation is used.

The current hand-rolled client is wire-correct by construction: it does zero runtime transform (JSON.stringify(req) / resp.json() as T) and shapes its interfaces to match sebuf's wire. es mode swaps that for canonical-shaped types plus an active codec that enforces canonical protojson — so it regresses exactly the protos annotations exist for.

Regression table — es mode vs current wire (Go server + hand-rolled TS)

"Current wire" is what both the Go server and today's TS client speak. ✅ = consistent, 🟡 = tolerated but annotation intent lost, 🔴 = broken (throws or silent data loss).

sebuf.http annotation	Current wire (Go + hand-rolled TS)	es mode (canonical protojson)	Status	Needed for drop-in
(none — plain scalars, nested msgs, repeated, maps)	canonical	canonical	✅	— works today
unwrap (root list/map, map-value)	[…] / {"AAPL":[…]}	{"users":[…]} (wrapped)	🔴 fromJson throws on array / shape mismatch	re-wrap on encode, unwrap on decode
oneof_config flatten + discriminator	{"type":"text","body":…}	{"text":{…}}	🔴 silent data loss (ignoreUnknownFields drops it)	flatten/inline + discriminator inject/extract
flatten / flatten_prefix	{"billing_street":…}	{"billing":{"street":…}}	🔴 silent data loss	promote/collapse nested keys w/ prefix
enum_value (custom string)	"active"	"STATUS_ACTIVE"	🔴 fromJson throws (unknown enum)	name↔custom-value map both directions
timestamp_format (unix/date)	1705312200 / "2024-01-15"	"2024-01-15T09:30:00Z"	🔴 fromJson throws (expects RFC3339)	re-encode/parse per format
bytes_encoding (hex/base64url/raw)	"48656c6c6f"	std base64	🔴 wrong/throws on decode	transcode to/from configured encoding
nullable (explicit null)	null vs absent vs value	null≡absent (protojson)	🟡 null/absent distinction lost	preserve explicit null on encode
empty_behavior (PRESERVE/NULL/OMIT)	{} / null / omitted	canonical (omit/{})	🟡 NULL/OMIT intent not honored	apply per-field empty policy on encode
enum_encoding=NUMBER (no custom value)	1	name string on encode, accepts number on decode	🟡 tolerated on decode; wrong on encode	emit number when annotated
int64_encoding=NUMBER	12345 (number)	"12345" (string)	🟡 tolerated (protojson accepts both)	emit number when annotated

Net: 6 annotations are outright broken, 4 more silently drop their intent. The wire-conformance test only exercises the ✅ row (defaults filled, unknown fields ignored), so none of this is caught today.

What "drop-in replacement" requires

To match Go, the es transport needs the TS analog of MarshalJSONSebuf: an annotation-aware shim layered over toJson/fromJson that rewrites the canonical JSON into sebuf's wire (and back) — flatten oneofs, apply enum_value maps, unwrap, transcode timestamps/bytes, honor null/empty policy. protoc-gen-es types don't carry sebuf annotations, so the sebuf plugin has to emit these transforms per message from the descriptors it already reads. That's real scope — effectively porting the internal/httpgen/*.go custom-JSON pipeline to TS.

Given that, I'd suggest landing this in two steps:

1. Now — make es mode safe by construction: fail loud. Extend the checkNoEnumParamsES pattern into a full guard that walks each RPC's transitive input/output/SSE-event message closure and errors at generation time if any field/message/oneof carries a JSON-mapping annotation es can't honor (all 🔴/🟡 rows above). The annotations package already exposes every getter needed (IsRootUnwrap, FindUnwrapField, GetOneofDiscriminatorInfo, GetEnumValueMapping, IsTimestampField, GetBytesEncoding, IsNullableField, GetEmptyBehavior, GetInt64Encoding, GetEnumEncoding). This ships a correct es mode restricted to protos it can actually round-trip.
2. Later — the transform layer, if/when we want annotated protos supported, tracked as a follow-up.
3. Docs — correct the framing now. "Known limitations" omits this, and the prose currently claims the opposite ("Go-server parity", "both directions go through protobuf-es's canonical JSON"). It should say plainly: es mode speaks canonical protojson and honors no sebuf.http JSON-mapping annotation; it is only wire-compatible with a sebuf server for protos that use none of them (enforced by the guard in step 1).

Two smaller correctness issues (independent of the above)

• Non-string path params emit uncompilable es code. Path params are assigned the raw URL string (tsservergen/generator.go:629, :649); for a numeric/bool field the branded type is number/bigint/boolean → TS2322. The get_params golden only covers a string path param. Fold this into the same guard (or coerce).
• Repeated non-string query params. req.x.forEach(v => params.append("p", v)) (tsclientgen/generator.go:467) lacks String(v), so repeated int32/bool won't typecheck. Add String(v).

Caveat on this report

This is a preliminary report: the wire shapes above are derived from reading the Go marshal pipeline and the protobuf-es codec, not from an executed cross-runtime round-trip. The exact throw-vs-silent behavior for the 🟡 rows in particular should be confirmed empirically. More broadly — this is the gap that matters most: if we want to trust this system, es mode needs real correctness checks, i.e. a conformance harness that round-trips each annotation through an actual sebuf Go server (or the hand-rolled client's wire) and asserts byte-equivalence, rather than only asserting protobuf-es's canonical behavior in isolation. That harness is also what would let us safely lift the fail-loud guard later, annotation by annotation.

Requesting changes on the wire-incompatibility blocker (fail-loud guard + docs at minimum). Happy to pair on the guard — it's a small, contained addition that turns this from an incompatible second wire format into a safe, honest opt-in.

[hishamank]

Update — addresses the "fail-loud + docs" blocker, plus the two smaller correctness issues

Pushed in 7c84726 and bed82f3.

Wire-incompatibility blocker → fail loud (internal/tscommon/es_guard.go)

CheckESMessageAnnotations walks each RPC's transitive request / response / SSE-event message closure (cycle-safe, through message fields and map values) and errors at generation time if any field/message/oneof carries a JSON-mapping annotation es-mode can't honor — the full 🔴/🟡 set: unwrap, oneof_config, flatten/flatten_prefix, enum_value, timestamp_format, bytes_encoding, nullable, empty_behavior, enum_encoding=NUMBER, int64_encoding=NUMBER. Wired into both generators alongside checkNoEnumParamsES. The error names the exact field + annotation, e.g.:

ts_runtime=protobuf-es: field User.status uses the enum_value JSON-mapping annotation (reachable from the response of UserService.GetUser), which es-mode cannot honor…

Regression tests in both generators run all ten annotation fixtures and assert the right annotation is rejected.

The two smaller correctness issues

• Non-string path params (server) — coerced from the raw URL string to the protobuf-es field type at both emission sites (create() init shape and branded-body merge): Number() / BigInt() / === "true" / string verbatim. Previously emitted TS2322.
• Repeated non-string query params — client wraps elements in String(v); server coerces the getAll() string[] to number[]/bigint[]/boolean[].

The get_params es fixture now covers numeric/bool path params (GET via create() and POST body-merge) and repeated string/int32/int64/bool query params, so the es golden tsc --noEmit typecheck (against real @bufbuild/protobuf@2.12.1) compiles all of it.

Docs framing corrected

Replaced the implied Go-parity wording with an explicit wire-contract note (es-mode speaks canonical protojson, honors no sebuf.http JSON-mapping annotation, is only wire-compatible for protos using none — enforced by the guard), added it as the headline Known limitation, and corrected the URL-param note.

Roadmap for lifting the guard (now in docs/client-generation.md)

es-mode is a preview. The staged path to a true drop-in for annotated protos:

1. Conformance harness — cross-runtime byte-equivalence test per (message, annotation), using the existing internal/httpgen/*_consistency_test.go as the oracle. Gate for lifting the guard.
2. Transform layer — the TS analog of MarshalJSONSebuf, as a single runtime shim over toJson/fromJson driven by a per-field annotation table (protobuf-es exposes the Schema at runtime, so no per-message codegen).
3. Lift the guard one annotation at a time, each only after green conformance. Suggested order: int64_encoding → enum_value → enum_encoding → timestamp_format → bytes_encoding → nullable → empty_behavior → flatten/unwrap/oneof_config.

Hand-rolled (default) output is unchanged except the repeated-query String(v) fix (runtime-identical for working code).

[hishamank]

Heads-up on review order: a follow-up PR #210 (feat/ts-es-result-errors) is stacked on top of this branch — it adds an opt-in ts_error_handling=result mode that makes es-mode unary clients return a typed Result union instead of throwing. It targets feat/ts-es-transport, so it should be reviewed/merged after this PR. Nothing to do here; just flagging the dependency.

[SebastienMelki]

Great work on this, Hicham — the architecture is clean, the fail-loud guards are thorough, and the hand-rolled default is byte-identical (zero regressions). We're very close to merge, just one required fix.

---

Required

Bug: single-value int64/uint64 query params not coerced in es-mode (server)

internal/tsservergen/generator.go:929-937 — the single-value query param path uses TSScalarTypeForField() which returns TSString for int64 (the hand-rolled representation). In es-mode protobuf-es expects bigint, but the code falls through to the default case and emits params.get("x") ?? "" — assigning a string where bigint is expected.

The repeated query param path (line 898) and path params (esPathParamInitExpr) both handle this correctly; the single-value path is the only one missing the es-mode branch.

Fix: add an es-mode check before the TSScalarTypeForField switch (analogous to the repeated field handling at line 898), reusing esPathParamInitExpr for the coercion. Then add a single-value int64 query param to get_params.proto so the es golden typecheck locks it in.

---

Suggestions (non-blocking, good follow-ups)

1. Expand wire-conformance test. The current conformance fixture covers omitted zero values + unknown field rejection — the core value prop. But it's narrow: no large int64 values (precision boundary), no nested messages with mixed populated/omitted fields, no map fields. Especially worth testing int64 boundary values since es-mode uses bigint vs hand-rolled string.

2. Add server-side wire conformance. The client has a wire-conformance test proving fromJson materializes defaults. The server has no equivalent proving toJson(create(...)) produces correct wire output. A symmetric test (encode a known message, assert the JSON matches a canonical fixture) would close that gap.

---

What looks good

• Fail-loud guards are comprehensive — JSON-mapping annotations, enum path/query params all rejected at generation time with clear errors.
• Emission-site import tracking is the right fix for the regex scanning bug. NeedProtobufES is structurally safe through EmitContext.
• es golden typecheck (tsc --noEmit with noUnusedLocals) is real CI protection against protoc-gen-es naming drift.
• Documentation is excellent — wire contract, MessageInitShape caveats, known limitations, and the roadmap (conformance harness → transform layer → annotation-by-annotation lift) are all clearly laid out.
• No regressions — all pre-existing hand-rolled goldens unchanged.

Thanks for the thorough work here — once the int64 query param fix lands this is ready to go. 🚀

[github-actions]

🔍 CI Pipeline Status

❌ Lint: failure
✅ Test: success
✅ Coverage: success
⚠️ Build: skipped
⚠️ Integration: skipped

---

📊 Coverage Report: Available in checks above
🔗 Artifacts: Test results and coverage reports uploaded

[codecov]

Codecov Report

❌ Patch coverage is 23.80952% with 112 lines in your changes missing coverage. Please review.
✅ Project coverage is 5.17%. Comparing base (b95aec5) to head (fa89fe6).

Files with missing lines	Patch %	Lines
internal/tscommon/es_guard.go	0.00%	56 Missing ⚠️
internal/tscommon/imports.go	44.06%	31 Missing and 2 partials ⚠️
internal/tscommon/types.go	0.00%	9 Missing ⚠️
internal/tscommon/modules.go	0.00%	8 Missing ⚠️
internal/tscommon/runtime.go	60.00%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##             main    #207      +/-   ##
=========================================
- Coverage   11.75%   5.17%   -6.58%     
=========================================
  Files          65      61       -4     
  Lines       11121   10412     -709     
=========================================
- Hits         1307     539     -768     
- Misses       9780    9848      +68     
+ Partials       34      25       -9     

Flag	Coverage Δ
unittests	5.17% <23.80%> (-6.58%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.