[Bug] Claude Code requests fail on Azure Opus when thinking parameters are not normalized per target

[xinyu-dev]

Environment

• Repo / commit: NVIDIA-NeMo/Switchyard @ 3acf3d86f10fead9f434b858dbbcf01a660b8db0
• Switchyard: 0.2.0
• OS: macOS 26.5.2
• Python: 3.13.5
• Claude Code: 2.1.220
• Key packages: anthropic==0.99.0, litellm==1.85.0
• Upstream: NVIDIA Inference Hub (https://inference-api.nvidia.com/v1)
• Models:
  • aws/anthropic/bedrock-claude-opus-5
  • azure/anthropic/claude-opus-5

Description

When Switchyard routes Claude Code traffic between the Bedrock and Azure deployments of Claude Opus 5, requests selected for Bedrock succeed while requests selected for Azure fail with HTTP 400.

Claude Code sends Anthropic thinking parameters using thinking.type=enabled. The Azure deployment requires thinking.type=adaptive and output_config.effort. Switchyard currently does not normalize this provider/model-specific difference after selecting the target, so an otherwise valid routing policy becomes partially unavailable.

The final launcher summary compounds the confusion because its per-model req value reports successful calls only, while failed attempts appear only in the aggregate error count. This can make it look as though Bedrock received all traffic even when Azure was selected and failed.

Steps to Reproduce

1. Create /tmp/opus-deployment-random.yaml:

   defaults:
     api_key: ${NVIDIA_INFERENCE_HUB_API_KEY}
     base_url: ${NVIDIA_INFERENCE_HUB_API_ROOT}
     format: openai
   
   routes:
     demo/opus-deployment-random:
       type: random_routing
       strong_probability: 0.5
       fallback_target_on_evict: bedrock
       strong:
         id: bedrock
         model: aws/anthropic/bedrock-claude-opus-5
       weak:
         id: azure
         model: azure/anthropic/claude-opus-5

2. Start Switchyard:

   source .venv/bin/activate
   switchyard serve --routing-profiles /tmp/opus-deployment-random.yaml --port 4000

3. In another terminal, route Claude Code through the proxy:

   ANTHROPIC_BASE_URL=http://127.0.0.1:4000 \
   ANTHROPIC_AUTH_TOKEN=switchyard \
   ANTHROPIC_API_KEY="" \
   ANTHROPIC_MODEL=demo/opus-deployment-random \
   ANTHROPIC_SMALL_FAST_MODEL=demo/opus-deployment-random \
   claude

4. Send several normal requests, such as Reply with only OK. For a deterministic comparison, temporarily set strong_probability: 1.0 to exercise Bedrock, then 0.0 to exercise Azure.

Actual Result

The Bedrock target succeeds. The Azure target returns:

400 litellm.BadRequestError: Azure_aiException -
{"type":"error","error":{"type":"invalid_request_error",
"message":"\"thinking.type.enabled\" is not supported for this model.
Use \"thinking.type.adaptive\" and \"output_config.effort\" to control
thinking behavior."}}

A representative final summary was:

requests : 20  (11 errors)
models:
  bedrock-claude-opus-5  9 req
  claude-opus-5          0 req

A minimal Azure request without Claude Code's thinking configuration succeeds, confirming that the deployment, model ID, credentials, and endpoint are valid.

Expected Result

Switchyard should normalize request parameters for the selected target so the same Claude Code request succeeds on both deployments.

For Azure Opus, enabled thinking should be represented as adaptive thinking, with supported effort configuration and without a fixed thinking budget. The session summary should also display per-model failures so failed selections are distinguishable from models that were never selected.

Suggested Fix

Add target-aware request normalization at the outbound backend boundary, after routing has selected the target:

• Rewrite unsupported thinking.type=enabled to thinking.type=adaptive for affected Azure Opus targets.
• Remove incompatible fixed-budget fields.
• Preserve supported effort through output_config.effort.
• Add tests that send the same Claude Code request through the Bedrock and Azure targets and assert the provider-appropriate outbound payload.
• Include each model's error count in the launcher session summary.

Target-level extra_body alone cannot reliably implement this because its current shallow merge is caller-wins and therefore cannot replace an inbound thinking field.

[fedecia]

Hit this from the opposite direction, and found a config-level workaround that needs no upstream change.

Same error, mirrored

Your case: Claude Code sends thinking.type=enabled, Azure Opus 5 requires adaptive + output_config.effort.

My case: Claude Code sends thinking.type=adaptive and output_config.effort=medium already — exactly what the error message asks for — and Switchyard's translation layer turns it back into enabled before it reaches Bedrock Opus 5. Same error string, generated rather than passed through:

400 bedrock error: The model returned the following errors:
"thinking.type.enabled" is not supported for this model.
Use "thinking.type.adaptive" and "output_config.effort" to control thinking behavior.

So this is not only "Switchyard doesn't normalize an inbound value". A conforming inbound request is rewritten into a non-conforming one. That widens the bug: a client can't avoid it by sending the right thing.

Environment

• switchyard-server 0.2.0 (crates.io, the only published version)
• macOS 26.6, Claude Code as the client, stage_router (not random_routing)
• Upstream: a PortKey gateway deployment → Bedrock, global.anthropic.claude-opus-5
• Efficient tier DeepSeek-V4-Flash, capable tier Opus 5

Different gateway, different route type, different provider — same defect, which suggests it sits in the translation layer rather than in any one provider adapter.

The trigger is output_config, not the inbound thinking value

Bisected against one captured Claude Code body (115 tools, thinking={"type":"adaptive","display":"omitted"}, output_config={"effort":"medium"}), replayed through the router:

Request	HTTP	Result
unmodified	400	"thinking.type.enabled" is not supported
thinking removed, output_config kept	400	same error
output_config removed, thinking kept	200	Opus 5 answers
both removed	200	Opus 5 answers

Removing thinking alone does not help. Removing output_config does. So the thinking: {"type": "enabled"} is synthesized from output_config.effort during translation, independent of what the client sent.

Control: the identical body sent directly to the same gateway and model — bypassing Switchyard — returns 200, streaming and non-streaming, with and without anthropic-beta: effort-2025-11-24. The payload is valid; only the proxied path fails.

What the debug trace shows

RUST_LOG=debug names the cause on the failing leg:

libsy.upstream_attempt{model="global.anthropic.claude-opus-5" wire_format=openai_chat ...}
libsy.client_call{... gen_ai.request.max_tokens=32000
                      gen_ai.request.reasoning.level="medium"
                      switchyard.routing.tier="strong" ...}

Two things stand out:

• wire_format=openai_chat — an Anthropic inbound request (/v1/messages) is being encoded as OpenAI Chat for the upstream, because that leg's client declares format = "openai_chat".
• gen_ai.request.reasoning.level="medium" — output_config.effort has become an OpenAI-style reasoning level. Re-encoding that back to Anthropic for Bedrock is where thinking.type=enabled appears.

Anthropic → OpenAI IR → Anthropic is not round-trip safe for the effort/thinking pair.

Workaround: declare the capable leg anthropic_messages

Declaring the Anthropic upstream with its native wire format skips the translation entirely:

[llm_clients.portkey_bedrock]
format = "anthropic_messages"   # was: openai_chat
base_url = "https://<portkey-gateway>/v1"

With that one change, the unmodified Claude Code body returns 200:

x-model-router-selected-model: global.anthropic.claude-opus-5
x-model-router-rationale: fall-through selected global.anthropic.claude-opus-5 (confidence 0.762)

A full stage_router run then interleaves correctly with total_errors: 0 — cheap turns on DeepSeek, escalated turns on Opus 5, and de-escalation afterwards.

Your repro config uses format: openai for both targets, so this may clear your Azure leg too — worth trying before waiting on a fix. It only applies when the upstream genuinely speaks Anthropic Messages; a mixed-provider route still needs per-target normalization, which is what this issue asks for.

One caveat for anyone taking the workaround on a gateway: switching that client to anthropic_messages changes how api_key_env is sent (as x-api-key). A gateway expecting its own auth header will start returning 401 — for PortKey the key has to move to extra_headers as x-portkey-api-key. Related: #360, #419, #420.

Confirming your observation about the summary counters

the per-model req value reports successful calls only, while failed attempts appear only in the aggregate error count

This bit me too, and it is worth fixing alongside. GET /v1/stats showed calls for the capable target while every one of those calls had 400'd, and the tiers array was empty, so nothing indicated the escalation had failed. I initially concluded the router was escalating successfully. The only reliable per-turn signals were the x-model-router-rationale header and the per-model errors count.