Skip to content

feat(ai-openrouter): support streaming structured output (response_format json_schema with stream: true) #526

@tombeckenham

Description

@tombeckenham

Summary

OpenRouter natively supports combining response_format: { type: 'json_schema', ... } with stream: true, but our adapter currently performs structured output as a separate non-streaming call. This means a logical structured-output run produces two distinct HTTP requests (one streaming free-text, one non-streaming JSON), and callers can't consume the JSON incrementally.

OpenRouter docs: https://openrouter.ai/docs/guides/features/structured-outputs#streaming-with-structured-outputs

Prior discussion: #370 (also related: #195)

Current behavior

packages/typescript/ai-openrouter/src/adapters/text.ts

  • chatStream() (line 113) — stream: true, no responseFormat. Free-text only.
  • structuredOutput() (line 266) — stream: false, sends responseFormat: { type: 'json_schema', jsonSchema: { name, schema, strict: true } }. Returns a Promise, no streaming.

The TextAdapter base interface (packages/typescript/ai/src/activities/chat/adapter.ts:97) hardcodes structuredOutput as Promise<StructuredOutputResult<unknown>>, so today there is no surface for incremental structured output even where the provider supports it.

Proposal

Add a streaming structured-output path. Two shapes worth considering:

  1. New adapter method structuredOutputStream(options) returning AsyncIterable<StreamChunk>, with chunks carrying partial JSON deltas (we already depend on partial-json for the same purpose elsewhere). Adapters that don't natively support it can fall back to the existing non-streaming structuredOutput.
  2. Extend chatStream to accept an outputSchema option and have OpenRouter emit responseFormat alongside stream: true. Simpler wire change, but couples structured-output semantics into the general chat path and makes the typed result harder to expose.

(1) is probably cleaner and parallels the chatStream / structuredOutput split we already have.

Acceptance criteria

  • @tanstack/ai-openrouter exposes streaming structured output that emits a single HTTP request with stream: true + response_format: json_schema (strict).
  • Partial JSON deltas are surfaced as they arrive (parsed via partial-json).
  • Final result validates against the supplied schema (re-using convertSchemaToJsonSchema(..., { forStructuredOutput: true }) for the strict transform).
  • E2E coverage in testing/e2e (mandatory per CLAUDE.md) — fixture + spec under the structured-output scenarios, with OpenRouter listed in feature-support.ts / test-matrix.ts.
  • Other adapters: at minimum, default to the existing non-streaming path so the new method is callable everywhere without breaking changes. Native streaming JSON for OpenAI/Anthropic/Gemini can land separately.

Notes / risks

  • Tool calls + structured output in the same request: OpenRouter's docs allow it but behavior varies by upstream provider — worth a smoke test, not a blocker.
  • Strict schema transform already lives in ai; reuse it, don't fork.
  • The base TextAdapter interface change is a breaking surface for downstream adapters — mitigated if we add structuredOutputStream as optional with a default that wraps structuredOutput.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions