Skip to content

[bug] --dry-run accepts an invalid llm client base_url #398

Description

@ting-hong-shieh

Symptom

On current main, a deployment with base_url = "not a url" passes
switchyard-server --dry-run, prints server OK: test-route, and exits 0. The
same deployment also starts normally; its first model request then fails with
HTTP 502 because the upstream request cannot be constructed.

Reproduction

No API key or live upstream is required.

Create routes.toml:

schema_version = 1

[llm_clients.invalid]
format = "openai_chat"
base_url = "not a url"

[targets.invalid]
id = "upstream-model"
llm_client = "invalid"

[routes.invalid]
id = "test-route"
type = "passthrough"
target = "invalid"

Run the dry check:

cargo run --locked -p switchyard-server -- \
  --config routes.toml --dry-run

It exits 0 and prints:

server OK: test-route

Start the same deployment:

cargo run --locked -p switchyard-server -- \
  --config routes.toml --host 127.0.0.1 --port 43173

Then send a request from another terminal:

curl -i http://127.0.0.1:43173/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"test-route","messages":[{"role":"user","content":"hello"}]}'

The response is:

HTTP/1.1 502 Bad Gateway

{"error":{"code":"upstream_configuration_error","message":"failed to build upstream request: builder error","type":"upstream_error"}}

Expected vs. actual

  • Expected: --dry-run rejects llm_clients.invalid.base_url as an invalid
    absolute HTTP(S) URL and identifies the affected client. This check should not
    require connecting to an upstream provider.
  • Actual: The non-empty value passes configuration validation. Its invalid
    syntax is discovered only when the first request is sent.

Environment

  • Switchyard commit SHA: b256d936f1d77bf13ec9bec399ea0a253e07ca05
  • Python version: N/A (Rust server path)
  • Rust version: rustc 1.96.1 (31fca3adb 2026-06-26)
  • OS / arch: macOS 26.5.2 / arm64
  • Install path: source build with cargo run --locked
  • Inbound format: OpenAI Chat Completions
  • Backend: none; the malformed URL prevents an upstream request from being built

Additional context

build_backend
currently checks only whether base_url is empty. The URL reaches
reqwest::Client::post
when a model call is made.

A local syntax check during deployment construction would catch the error while
preserving --dry-run's no-network behavior. Validating the resolved endpoint
with the same URL-joining rules used at request time would also avoid creating a
second interpretation of base_url.

I searched open and closed issues and pull requests for base_url, dry-run,
invalid URL, builder error, and upstream_configuration_error, and found no
matching report.

I'm happy to send a PR for this if the expected validation behavior sounds right.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions