HybridInference is a FastAPI gateway that speaks the OpenAI/OpenRouter HTTP API and dispatches each request to one of several interchangeable upstreams: a local inference server (vLLM, SGLang, Ollama, or anything else OpenAI-compatible) or a hosted API. The point of the gateway is that a model id the client asks for is decoupled from the endpoint that actually serves it, so the same client call can be load-balanced, failed over, priced, and logged across a mix of machines you own and machines you rent.
The backend is one Python package tree under apps/backend/, split in two:
| Directory | Responsibility |
|---|---|
apps/backend/serving/ |
HTTP surface, authentication and quota, request schemas, provider adapters, storage, observability |
apps/backend/routing/ |
Route table, router strategies, endpoint health, circuit breaker, fallback |
Both are importable as top-level packages (serving.*, routing.*); the import
root is apps/backend, declared in pyproject.toml.
Five ideas explain most of the code:
- One model id, many endpoints. Clients name a model; the gateway owns which machine serves it. Everything else follows from that.
- Failure is routed around, not retried into. A non-idempotent generation is never re-sent with the same key; resilience comes from rotating keys, the fallback chain and the circuit breaker.
- A caller learns nothing it is not entitled to. Access failures collapse
to a uniform
404, and upstream errors are scrubbed before they reach a client. - Configuration is a deployment's, not the project's. The repository ships runnable examples; a real deployment configures its registry, routing policy and branding in Admin and PostgreSQL. See Configuration.
- Extension points are declarative. New provider: a
kindand, if the API is not OpenAI-compatible, an adapter. New routing behaviour: a module inapps/backend/routing/strategies/that self-registers. Neither requires touching the request handler.
┌──────────────────────────────────────────┐
HTTP client ────▶│ Serving layer apps/backend/serving/ │
(OpenAI SDK, │ │
Anthropic SDK, │ middleware → auth/quota → model gate │
curl, IDE) │ servers/app.py, servers/auth.py, │
│ servers/routers/completions.py │
└────────────────────┬─────────────────────┘
│ model id + messages
▼
┌──────────────────────────────────────────┐
│ Routing layer apps/backend/routing/ │
│ │
│ per-model router → weighted selection │
│ circuit admission → automatic fallback │
│ routers.py, model_router_registry.py, │
│ endpoint_health.py │
└────────────────────┬─────────────────────┘
│ chosen adapter
▼
┌──────────────────────────────────────────┐
│ Adapter layer │
│ apps/backend/serving/adapters/ │
│ │
│ translate request/response, own the │
│ API key, normalize usage + errors │
└────────────────────┬─────────────────────┘
│ HTTPS
▼
┌──────────────────────────────────────────┐
│ Providers: local vLLM / SGLang / Ollama, │
│ OpenRouter, Anthropic, Gemini, any │
│ OpenAI-compatible service │
└──────────────────────────────────────────┘
Alongside every layer, both under apps/backend/serving/:
storage/ Postgres operational store + request log (api_logs)
observability/ structured logs, alert rules, Slack alerting
For how a public deployment gets traffic to the gateway — reverse proxy, CDN, and the console's own path rewrites — see The public path table. Nothing in this page depends on that topology; the gateway is an ordinary HTTP server.
The chat-completions path in
apps/backend/serving/servers/routers/completions.py is the canonical one; the
other inference surfaces reuse its parts.
create_app() in apps/backend/serving/servers/app.py registers nine
middlewares. Starlette runs the last-registered outermost, so the effective
order from the outside in is:
NoSniffMiddleware— addsX-Content-Type-Options: nosniffto every response that lacks it, touching only the response start.RequestIdMiddleware— mints or adopts a request id used by every log line.FallbackErrorMiddleware— last-resort error shaping.KubernetesProxyTrustMiddleware— discovers which proxies may report the client address, when Kubernetes proxy discovery is configured.RequestLogMiddleware— one structured log record per HTTP request.TimeoutMiddleware—REQUEST_TIMEOUT_SECONDS(default 120s) for ordinary requests,STREAM_REQUEST_TIMEOUT_SECONDS(default 3600s,<=0disables the cap) once a response has been marked as streaming.CORSMiddleware.TrailingSlashRedirectMiddleware— answers a path under/v1or/anthropicthat ends in a slash with a308to the bare path without it, as the console does; see The public path table.RequestBodyLimitMiddleware— answers413to a request body larger thanREQUEST_MAX_BODY_MB(default 32 MiB): before reading anything whenContent-Lengthdeclares more, otherwise as soon as the bytes received pass the limit. Batch input uploads keep their own limit,BATCH_MAX_BODY_MB. Behind the console, every gateway path that takes a request body streams through a route handler and is bounded only here, so raisingREQUEST_MAX_BODY_MBtakes effect behind the console too.
Accepted connections have one more clock, and it is not a middleware.
Uvicorn arms its keep-alive timer only while it is waiting for the next
request, and cancels that timer on the first byte. A client that then stops,
including one that never finishes the headers, holds the socket until the
process restarts. HTTP_IDLE_TIMEOUT_SECONDS (default 600, <=0 disables
it) closes that connection after that many seconds with no bytes in either
direction. Response bytes reset it, so a stream that keeps sending is limited
by the streaming clocks below rather than by this one. It is longer than
REQUEST_TIMEOUT_SECONDS and the inter-chunk stream idle, so a slow prefill
still receives the gateway's 504 instead of a dropped connection.
Control routes get a much smaller body limit: 256 KiB under /auth/, /user/,
/internal/ and /v1/identity/ and on /v1/batch_webhook, 1 MiB under
/admin/. FastAPI reads and decodes a route's declared body model before any
dependency runs, authentication included, so an anonymous caller's body is
decoded before the 401. The routes that legitimately take a larger body (a
configuration restore, the playground, the alert-history ingest) and
/v1/embeddings read theirs only once the caller is authenticated, and keep
REQUEST_MAX_BODY_MB.
Middleware order matters: streaming responses are StreamingResponse
objects that must not be buffered. Any new middleware that reads a response body
before forwarding it will break Server-Sent Events. The body limit counts request
bytes as they pass and holds none of them.
verify_api_key in apps/backend/serving/servers/auth.py is a FastAPI
dependency on every inference endpoint. It resolves, in order:
- Agent grant tokens — a separate credential namespace, checked first.
A grant may only be used on inference paths; anything else gets
403 insufficient_scope. - Auth disabled — when
USER_AUTH_ENABLEDis falsy the caller is treated as an anonymous admin. The setting is fail-closed: auth is on unless explicitly turned off. - API key — keys are minted as
hyi-<url-safe token>and stored only as an HMAC-SHA256 digest keyed byAPI_KEY_SECRET. The key may arrive asAuthorization: Bearer …orX-API-Key.
The same dependency enforces the caller's daily spend quota and returns 429
with a shared payload builder (apps/backend/serving/quota.py) whichever door
the request came through. A separate enforce_user_concurrency dependency holds a
per-user in-flight slot for the duration of the request. Which budget the slot
comes from depends on the model (a concurrency-exempt model has its own), so the
body is read before the slot is taken; a caller may have four bodies read that
way at once, and a fifth request gets the same 429 before its body is read.
The spend the gate reads comes from the counter tables (user_daily_cost,
api_key_daily_cost), and the spend ledger (apps/backend/serving/spend_ledger.py)
adds what those tables do not show yet:
- Requests in flight.
enforce_user_concurrency(and the batch worker, for each item) checks the quotas and the key's budgets again and reserves the request's estimated cost in one step, then releases the reservation when the request ends, however it ends. A request is refused only when not evenESTIMATED_REQUEST_COST_USDfits, as at the gate. An online request then reserves what it may cost (its prompt and itsmax_tokensat its model's prices, at mostMAX_RESERVED_REQUEST_COST_USD), or what its limits have left if that is less; a batch item reservesESTIMATED_REQUEST_COST_USD. Requests admitted at the same time therefore cannot all fit into the same remaining budget. - Costs not yet written. A finished request's cost is recorded in the
ledger and written to the counters by one writer per store: costs recorded
while a write is in flight are coalesced per user, key and UTC day into the
next, which goes out as one statement, account and key counters together.
The ledger keeps counting a cost until its write is confirmed (and for a
while after, for a gate whose read began before it landed), so the delay
never lets a request past a limit. A failed write is retried; shutdown
writes what is left and logs anything it could not. An entry the counters
refuse for its own values, such as one that would overflow its counter, is
set aside rather than retried, so it cannot hold back other accounts' costs:
it stays counted in the process and is logged (
cost_counter_write_refused) and alerted, for an operator to reconcile. One request adds at most $1,000; a larger cost is recorded as $1,000, logged (cost_above_request_ceiling) and alerted.
The ledger is per process. With several worker processes each sees only its own requests in flight and costs not yet written, so the limits are protected within each process, not across them; production runs one worker.
A response the client closes early, or that the gateway's request timeout cuts
off, is billed for what the upstream already did, streamed or not: the usage it
reported, or else an estimate of the prompt and of the output so far
(metadata.usage_estimated marks the estimate). A non-streaming request is
billed so only once it has left the gateway; one still waiting for an outbound
slot was never seen upstream. A request the upstream fails is not billed.
Read-only endpoints such as /v1/models use optional_verify_api_key instead:
a missing or invalid key is anonymous rather than rejected, and the key only
decides which models are visible.
Before any provider is contacted, the handler rejects the request with 404 if
the model is not in the route table, is unpublished, requires a role the caller
does not have, was disabled for that user, or is outside a grant's scope. All
five conditions return the same "model not found" body on purpose, so a caller
cannot enumerate models it is not entitled to by reading the difference between
403 and 404.
Each model resolves to a router instance through ModelRouterRegistry
(apps/backend/routing/model_router_registry.py), which reads the model's
router: / router_params: fields from the model registry and caches one
router per model id. A model with no router: uses the default_router value
from the routing config.
The router then picks one adapter for this request (FixedRouter._select_adapter):
- routes whose provider is admin-disabled carry weight 0 and are skipped;
- routes whose input modalities cannot accept the request's media are excluded;
- routes whose circuit is open are not admitted — if that leaves nothing, the
request fails with
AllCircuitsOpenError; - among the survivors, selection is weighted-random, biased away from endpoints already saturated with prefill work;
- a short-lived per-caller affinity (5 minutes) re-pins a conversation to the endpoint that already holds its prefix cache, unless that endpoint has built up a backlog.
An admin may bypass selection entirely with an X-Route-Pin request header
naming a provider label or endpoint_id. A pinned request never falls back —
a silent switch would make the pin meaningless.
The router then calls the chosen adapter, through a thin wrapper that holds exactly the adapter it chose, so an admin edit made in the meantime cannot redirect a request that is already on its way. See Routing Internals.
On success the endpoint is recorded healthy and the response carries an
internal _routing block (provider, base URL, endpoint_id).
On failure the endpoint records a failure and — unless the caller pinned a
provider — the router walks the model's remaining route legs in route order,
skipping legs that are disabled, modality-incompatible, or circuit-open, and
tries each in turn. Every attempt is appended to a failed_attempts list that
travels with the eventual response or error, so the request log attributes the
failure to real upstreams rather than to the router. If every leg fails, the
primary error is re-raised.
Circuit state lives per endpoint_id in
apps/backend/routing/endpoint_health.py:
| Knob | Env var | Default |
|---|---|---|
| Consecutive failures that open a circuit | CIRCUIT_FAILURE_THRESHOLD |
3 |
| Seconds an open circuit waits before a half-open probe | CIRCUIT_COOLDOWN_SECONDS |
30 |
| Availability floor | CIRCUIT_MIN_AVAILABILITY |
0.7 |
| EWMA smoothing for the availability estimate | ROUTER_HEALTH_EWMA_ALPHA |
0.1 |
| Seconds a caller's repeat failures stay off a circuit that has served others since its counted failure | CIRCUIT_CALLER_WINDOW_SECONDS |
60 |
Client errors do not trip the breaker: a 4xx other than 408, 429, 401, and 407 is the caller's problem, and counting it would let one malformed request take an endpoint away from everybody. 408 and 429 signal upstream overload and do count; 401 and 407 are unambiguous rejections of the gateway's own credential and count too, because no user can fix them.
Nor can one caller open a circuit by itself while the endpoint keeps serving
other callers: its failure there counts, and its further failures, until it is
served again, are skipped for up to CIRCUIT_CALLER_WINDOW_SECONDS if the
endpoint has served somebody else since. An outage serves nobody, so it still
opens the circuit at once. See
Endpoint health and circuit breaking.
Separately, the routing file can turn on a probe that polls the /health
path of the local endpoints it lists. The probe only reports; it never changes
routing. See The routing file.
After the response is produced, CompletionsLogger
(apps/backend/serving/servers/routers/completions_logging.py) schedules two fire-and-forget
side effects: a row in api_logs, and a RoutingObservation handed back to the
router. FixedRouter ignores observations; online-learning routers use them to
update their cost model.
Diagrams of the routers and how they execute a request are in Routing Internals.
There are exactly three modules under apps/backend/routing/strategies/, and
they are not all the same kind of thing:
| Module | What it is |
|---|---|
fixed.py |
Registers the fixed strategy: weighted-random selection with automatic fallback, implemented by FixedRouter in apps/backend/routing/routers.py |
routewise.py |
Registers the routewise strategy: a cost-aware router that learns from RoutingObservation feedback, implemented under apps/backend/routing/routewise/ |
weight.py |
Not a per-model router at all. FixedRatioStrategy splits a weight budget between the local and remote endpoint groups; it is used by RoutingManager, not by the router registry |
fixed and routewise are the only values router: accepts. Selecting an
unregistered name fails configuration validation with a message listing the
known strategies. Each strategy declares a Pydantic params model with
extra="forbid", so a typo in router_params: fails at boot instead of
silently falling back to a default.
Two different things are called a strategy:
routing config default_router: fixed
│ (+ local_deployment / remote_deployment pools)
▼
RoutingManager ──uses──▶ FixedRatioStrategy → rewrites per-adapter WEIGHTS
(routing/manager.py) (strategies/weight.py) for models whose endpoints
appear in those pools
model registry router: fixed | routewise
│ router_params: {...}
▼
ModelRouterRegistry ──▶ build_router() → chooses WHICH ROUTER runs
(model_router_registry.py) for one model's requests
RoutingManager only rewrites weights when the effective strategy is fixed;
it returns without touching anything otherwise. Per-route weights declared in
the model registry already encode a local/remote split, so a deployment that
does not list endpoint pools in its routing config simply keeps its declared
weights. On a gateway with a database the rewritten weights are not used at
all; see The routing file.
Every router runs the endpoint it picked through a thin wrapper called a leaf.
An experimental composition, HybridRouter, can also divide a model's routes
into a local and a cloud pool and plan across the two; it is off unless a model
opts in. Routing Internals describes both, with
diagrams.
Two identifiers look similar and mean different things.
provider is a label on the route. It defaults to the adapter kind and can be
overridden per route with provider: in the model registry. It is what
api_logs.provider records, what per-provider dashboards group by, and what the
admin disable switch targets. Two local vLLM boxes can carry different provider
labels so their traffic stays in separate cohorts. A route may not borrow a
label that belongs to an adapter kind, including kinds added by backend
extensions.
endpoint_id is the unique key for one endpoint of one model, written
{model_id}:{location}:
- a host of
localhost,127.0.0.1,0.0.0.0orhost.docker.internalyieldslocal-<port>, orlocalwhen there is no port; - otherwise the location is derived from the kind or hostname, e.g.
<model>:openrouter-api.
Latency profiles, availability tracking, and circuit-breaker state are all keyed
on endpoint_id, which is why two route legs pointing at the same base URL with
different pinned upstreams still get independent breakers.
The suffix is not an ownership signal. A gateway-owned server on a LAN address
is stamped from its hostname like any remote service, and an admin-supplied
route_id becomes the endpoint_id verbatim. Do not infer "is this machine
mine?" from the string.
An adapter is the object that knows how to talk to one provider: it builds
the URL and headers, holds the API key (or a rotating pool of keys), translates
the request body and the response, normalizes usage accounting, and raises
errors in a shape the router understands. Adapters live in
apps/backend/serving/adapters/ and are constructed from the model registry by
_make_adapter in apps/backend/serving/servers/registry.py.
| Module | Class | Covers |
|---|---|---|
openai_compat.py |
OpenAICompatAdapter |
Every OpenAI-compatible service, including local vLLM, SGLang, and Ollama. Local inference servers have no dedicated adapter |
openrouter.py |
OpenRouterAdapter |
OpenRouter; see Routing through OpenRouter |
anthropic.py |
AnthropicAdapter |
The Anthropic Messages API directly |
claude.py |
ClaudeAdapter |
Claude served through Google Vertex |
gemini.py |
GeminiAdapter |
The Gemini API |
_make_adapter first checks the factory table populated by explicitly enabled
backend extensions. Without a registered
factory, it maps a route's kind: onto a built-in adapter and pre-seeds
provider-specific configuration — a usage profile, a non-standard chat path, or
whether it is safe to send stream_options: {include_usage: true}. An extension
factory instead receives the configuration dictionary before those defaults.
Adding a provider that is already OpenAI-compatible usually means selecting a
profile, not writing a new class; see Adding a New Model.
Key rotation is an adapter concern. When a route declares api_keys: (plural),
OpenAICompatAdapter draws from a pool: a key that hits a key-specific or
transient failure (429, 401/402/403, 408/425, 5xx, timeouts) hands the request
to the next key. Rotation comes before muting — while another key is still
untried the failing one keeps its place in the pool, and only a key the request
runs out of alternatives on is muted, for 20 seconds. Request-scoped errors such
as 400 and 422 fail on every key, so they propagate immediately instead of
burning the pool. A completion POST is never retried against the same key —
re-sending a non-idempotent generation would double-bill it. Resilience comes
from the router's fallback chain, not from blind retries.
An upstream that stops sending mid-generation while keeping the socket open is a fault the gateway has to notice on its own. Waiting for the connection to break means finding out only once something further out gives up — and by then every stream committed to that endpoint has already died together, while new ones kept being dispatched to it. Four clocks cover a streaming request, staggered so the innermost one reports first:
| Clock | Default | Covers |
|---|---|---|
STREAM_IDLE_TIMEOUT_SECONDS |
180s | Gap between two data-bearing SSE frames, after the first |
STREAM_MAX_IDLE_S |
240s | Same gap on /v1/messages, counted in forwarded frames |
STREAM_FIRST_BYTE_TIMEOUT_SECONDS |
unset | Connect to first frame, i.e. prefill |
STREAM_REQUEST_TIMEOUT_SECONDS |
3600s | The whole streaming response |
The first-byte and inter-chunk budgets are separate on purpose. A socket-level
read timeout (sock_read) restarts on every read, so it cannot give a long
prefill room without giving a stalled backend the same room. Prefill is
legitimately slow — a 1M-token prompt can take more than two minutes to produce
its first token on a self-hosted server — while the gap between chunks after
that is a few milliseconds. So only the inter-chunk budget carries a default; a non-positive value
disables either. /v1/messages keeps its own, looser ceiling because it counts
frames the gateway forwards, and a buffered XML tool call can hold one back for
a minute-plus without the upstream being idle at all.
When the inter-chunk budget expires, OpenAICompatAdapter raises
UpstreamStreamIdleError — distinct from the end-of-body
_INCOMPLETE_STREAM_ERROR, which is what a deployment proxy's own read timeout
looks like from here. It propagates as a stream_exception, so the endpoint
loses availability and its circuit opens, rather than the partial answer being
capped with a fabricated [DONE].
Application documents are validated JSON settings in app_config:
MODEL_CATALOG, ROUTING_POLICY, ALERT_RULES, SITE_CONFIGURATION,
SITE_BRANDING and SSO_PROVIDERS. Application YAML files and their path/mode selectors are not
runtime inputs. A fresh gateway serves an empty model list and a neutral console
that asks an administrator to configure it.
The model catalog supplies the baseline; operational provider, key, route and weight state is applied on top. Ordinary Admin route edits synchronize between backend processes through committed, model-scoped change counters. Existing routing transition helpers retain execution and feedback ownership. Per-process acknowledgements distinguish saved configuration from applied configuration. Catalog/routing-policy/alert-rule changes remain explicitly restart-required.
Credentials are separate secret settings referenced by documents. The existing
settings store owns validation, audit, conditional writes and backup/restore.
An explicit offline importer handles old external YAML; startup never imports
those files. See Configuration and the
{download}accepted design <../agents/specs/2026-11-08-database-only-configuration.md>.
Persistence is defined by two abstract base classes in
apps/backend/serving/storage/base.py:
OperationalStore— accounts, API keys, roles, quotas, admin-managed providers and routes, runtime settings.LogStore— the request log.
Postgres implements both (postgres_operational.py, postgres_log.py).
CachedOperationalStore wraps the operational store with an in-process cache,
because auth resolves against it on every single request.
The schema, the request log table api_logs and its rollups included, comes
from the versioned migrations in apps/backend/migrations/; the backend creates
no tables of its own. Columns of api_logs worth knowing: request_id, model_id, provider,
served_model_id / served_endpoint_id (which endpoint actually answered, as
opposed to what the client asked for), ttft_ms and latency_ms, the token
counts, cost_usd (what the caller is billed, from the model's own pricing) and
upstream_cost_usd (what the upstream reported, when it reports one).
A deploy applies the migrations before it restarts the backend, and the backend
refuses to start on a database they have not reached. Each migration statement
waits for a table lock only briefly: an ALTER TABLE that queued behind a long
query would block every reader behind it. See Database.
The gateway starts without a database. /health then reports
database_connected: false, and request logging and accounts are unavailable,
but routing and completions still work — which is what makes the router
tutorial's first stage runnable with no Postgres at all.
The gateway reports through these; it has no Prometheus exporter:
- Structured logs.
RequestLogMiddlewareemits one record per HTTP request;apps/backend/serving/utils/logging.pyshapes them and suppresses noise from health-probe paths unlessLOG_LEVEL=DEBUG. - The request log.
api_logsis the durable record and the source for the admin dashboards and usage reporting. - Health endpoints.
/healthis a liveness check that returns 200 withstatus: "degraded"when a configured store is down but traffic can still be served, and 503 only when every configured store is unreachable./health/readyis the strict variant: 503 unless everything configured is up. Both tell an anonymous caller onlystatus; the reason, instance, route count and stores are shown to an administrator./health/deepadds per-endpoint availability and circuit state, and reportsdegradedwhen any endpoint is. All three answer from one shared store probe for two seconds, and concurrent callers wait for the probe in flight, so polling them does not cost a database query per call. - Alerting.
alert_slack()andalert_on_transition()inapps/backend/serving/observability/alerts.pypost through anAlertSinkto one Slack or WeCom webhook, selected in Admin Configuration. Channel changes take effect after a restart. Thresholds come from theALERT_RULESdatabase document; an empty document uses built-in thresholds. The circuit breaker pages oncircuit_opentransitions and on upstream-credential rejections, with per-endpoint cooldowns so a persistent fault re-pages on a fixed cadence instead of flooding. - Alert history (on with a database;
ALERT_HISTORY_ENABLED=falseturns it off). The same alert path records what it did — breaches, recoveries, suppression decisions, deliveries and their results — into PostgreSQL through a bounded buffer and one writer per process (serving/observability/alert_history/). It never changes what is sent. See Alert history.
Client-facing groups, all served by the same app:
| Group | Paths | Auth |
|---|---|---|
| OpenAI-compatible inference | POST /v1/chat/completions, POST /v1/completions, POST /completion, POST /v1/embeddings, POST /v1/responses |
API key |
| Anthropic-compatible inference | POST /v1/messages, POST /anthropic/v1/messages, …/count_tokens |
API key |
| Batch inference | POST and GET /v1/batches, GET and DELETE /v1/batches/{id}, POST /v1/batches/{id}/cancel, GET /v1/batches/{id}/results, GET /v1/batches/{id}/output.jsonl (JSONL export); chat, embeddings, Responses and Messages targets, as direct requests or a JSONL file |
API key; creating a batch needs the BATCH_MIN_ROLE role, internal by default |
| Batch files | POST and GET /v1/files, GET and DELETE /v1/files/{id}, GET /v1/files/{id}/content |
API key; uploading a file needs BATCH_MIN_ROLE |
| Batch webhooks | GET, PUT and DELETE /v1/batch_webhook, GET /v1/batch_webhook/events |
API key; setting up or changing a webhook needs BATCH_MIN_ROLE |
| Model catalog | GET /v1/models, GET /models, GET /openrouter/models, GET /anthropic/v1/models |
Optional — a key only widens what is listed |
| Balance probes | GET /v1/usage, GET /v1/dashboard/billing/subscription, GET /v1/dashboard/billing/usage, GET /v1/dashboard/billing/credit_grants, GET /v1/sub2api/billing |
API key. A spent quota still answers; these routes do not admit work |
| API key description | GET /v1/key |
API key |
| Health | GET /health, /health/ready, /health/deep |
None; upstream addresses only for an admin |
| Public routing disclosure | GET /routing |
None; upstream addresses only for an admin |
| Admin | GET /admin/routing, GET /admin/stats and the rest of /admin/* |
Admin |
| Accounts and console APIs | /auth/*, /user/*, /site-config |
Mixed |
BATCH_ENABLED controls all batch, file and webhook routes together; when it
is off, even management and result routes answer 404, and no queued request
starts. With it on, all batch kinds are available. BATCH_MIN_ROLE controls
creating batches, uploading files and setting up or changing a webhook. An
owner whose role is lowered can still manage existing batches and files, read
or delete their webhook, and receive its deliveries.
GET /routing requires no authentication and returns each published route's
provider label and traffic weight. Its upstream base URL, and the routing
file's probe results, which are keyed by endpoint URL, are shown only to a
caller that sends an administrator's JWT or ADMIN_TOKEN in the
Authorization: Bearer ... header, and /health/deep leaves the base URL out
of its route_exclusions on the same terms. GET /admin/routing requires
those credentials and additionally includes unpublished routes.
/v1/models shape-shifts by client: an Anthropic-family client calling it
receives the Anthropic list response, while /models and /openrouter/models
always return the OpenAI/OpenRouter shape.
The balance probes answer the quota checks that new-api, one-api, sub2api and
cc-switch run on a timer or on demand. Four of them report the money left under
the tightest of the account's daily quota, its monthly quota when one is set,
and the presented key's and project's budgets; /v1/sub2api/billing answers
sub2api's rate-multiplier probe. A limit reads as spent once the request gate
would refuse a request on it, so remaining is 0 exactly when a request would
be refused, and a spent quota still answers HTTP 200. new-api disables a
channel whose balance check reads 0 or less when the channel's automatic
disabling is on, and enables it again only if its operator turned on automatic
enabling. While user auth is disabled the same paths report an unlimited
balance.
GET /v1/key describes the calling key: its project, models, budgets and
concurrency.max_in_flight, the effective in-flight cap the limiter enforces
for it, or null when unlimited. When a project cap is the tighter one, that
slot is shared across every key in the project, so a key can still receive
429 below max_in_flight while sibling keys hold the slot.
The repository ships Dockerfiles and Compose files under deploy/docker/:
Dockerfile.backend (the gateway), Dockerfile.frontend (the Next.js console
in apps/frontend/), and docker-compose.yml. Systemd units live in
deploy/systemd/.
The console and the API share one origin: the console's route handlers forward
/v1, /anthropic, /auth, /user, /admin and the published /internal
paths to the backend, and the Next.js rewrites in apps/frontend/next.config.js
forward /health and /site-config, so a browser session and an API key reach
the same paths on the same host. Those handlers and rewrites — not a reverse
proxy config — are the public path table. The frontend
has its own toolchain and quality gates, separate from the Python make
targets.
See Deployment Guide to run it, Installation for a local checkout, and Contributing before sending a change.