Client api-key and OpenAI org/project headers are forwarded upstream

[Atharva-Kanherkar]

Summary

RESERVED_HEADERS in crates/libsy-llm-client/src/client.rs strips authorization and x-api-key from forwarded client metadata, but does not strip three headers of the same class:

• api-key (Azure OpenAI's credential header, the x--less sibling of x-api-key)
• openai-organization (OpenAI billing/tenant selector)
• openai-project (OpenAI project selector)

switchyard-server's metadata_from_headers stores the entire inbound HeaderMap, and send_once runs forward_metadata_headers (copies every non-reserved client header) before apply_auth sets the backend credential. Because those three names are not reserved, a client-supplied copy rides upstream next to the backend's own key.

This is a trust-boundary gap: the point of the reserved list is that a caller cannot influence the credential or tenant identity used upstream. It works for authorization and x-api-key; it misses the three above.

Impact

• OpenAI backends: a client OpenAI-Organization (or OpenAI-Project) is forwarded. An invalid value makes the upstream return 401 mismatched_organization. A valid value that the backend key can access bills and attributes the request to that org or project instead of the intended one. In a multi-tenant deployment behind one shared key this lets one caller redirect billing or attribution within the key's scope.
• Azure OpenAI backends that authenticate with api-key: a client api-key is forwarded as the credential header. If the operator did not set api-key via extra_headers, the client-supplied value is what reaches upstream.
• On plain OpenAI and Gemini, api-key is ignored (the bearer wins), so there the observable effect is a client-controlled header leaving the proxy rather than an auth swap.

No operator secret is leaked to the client, and the org/project vector is bounded to tenants the backend key can already reach. Reported here with a fix; happy to route through psirt@nvidia.com instead if you prefer to treat it as a security report.

Affected

switchyard-server 0.2.0 and current main. The RESERVED_HEADERS list is identical in both.

Reproduction

Point a Switchyard passthrough route at a mock upstream that records inbound headers, configure the backend credential, and send the client headers.

Backend config (sy.toml):

schema_version = 1
[llm_clients.local]
format = "openai_chat"
base_url = "http://127.0.0.1:19999/v1"
api_key_env = "MOCK_KEY"
[targets.cap]
id = "captured-model"
llm_client = "local"
[routes.primary]
id = "captured-model"
type = "passthrough"
target = "cap"

Mock upstream that appends each request's headers to a file:

import json, sys
from http.server import BaseHTTPRequestHandler, HTTPServer
CANNED = {"id":"x","object":"chat.completion","created":0,"model":"captured-model",
          "choices":[{"index":0,"message":{"role":"assistant","content":"pong"},"finish_reason":"stop"}],
          "usage":{"prompt_tokens":1,"completion_tokens":1,"total_tokens":2}}
class H(BaseHTTPRequestHandler):
    def do_POST(self):
        self.rfile.read(int(self.headers.get("content-length", 0)))
        with open("captured.jsonl", "a") as f:
            f.write(json.dumps({k.lower(): v for k, v in self.headers.items()}) + "\n")
        b = json.dumps(CANNED).encode()
        self.send_response(200); self.send_header("content-type","application/json")
        self.send_header("content-length", str(len(b))); self.end_headers(); self.wfile.write(b)
    def log_message(self, *a): pass
HTTPServer(("127.0.0.1", 19999), H).serve_forever()

Run:

python3 mock.py &
MOCK_KEY="OFFICE_KEY" cargo run -p switchyard-server -- --config sy.toml --port 19000 &

BODY='{"model":"captured-model","messages":[{"role":"user","content":"ping"}]}'
curl -s -o /dev/null localhost:19000/v1/chat/completions -H 'content-type: application/json' -H 'OpenAI-Organization: CANARY_ORG' -d "$BODY"
curl -s -o /dev/null localhost:19000/v1/chat/completions -H 'content-type: application/json' -H 'OpenAI-Project: CANARY_PROJECT' -d "$BODY"
curl -s -o /dev/null localhost:19000/v1/chat/completions -H 'content-type: application/json' -H 'api-key: CANARY_AZURE_KEY' -d "$BODY"
curl -s -o /dev/null localhost:19000/v1/chat/completions -H 'content-type: application/json' -H 'x-api-key: CANARY_X_API_KEY' -d "$BODY"

Observed in captured.jsonl (repeated 5/5 per header):

Client header	Reaches upstream?
OpenAI-Organization: CANARY_ORG	yes
OpenAI-Project: CANARY_PROJECT	yes
api-key: CANARY_AZURE_KEY	yes
x-api-key: CANARY_X_API_KEY	no (stripped, as intended)

The upstream authorization stays Bearer OFFICE_KEY in every case, confirming the leak is the extra header riding alongside the backend credential.

Fix

Add api-key, openai-organization, and openai-project to RESERVED_HEADERS so forward_metadata_headers drops the client copy. An operator that legitimately needs one of these upstream still sets it through the backend's extra_headers, which are applied after forwarding and therefore unaffected. PR with the change and a regression test follows.