Skip to content

serving.md

Latest commit

 

History

History
690 lines (574 loc) · 37.6 KB

serving.md

File metadata and controls

690 lines (574 loc) · 37.6 KB
Raw

HTTP serving

build/apps/ninfer-serve loads one registered artifact and exposes OpenAI- and Anthropic-compatible HTTP endpoints over one resident NInfer Engine.

Start the server

./build/apps/ninfer-serve models/qwen3_6_27b.ninfer \
  --host 127.0.0.1 \
  --port 8080 \
  --max-context 16384 \
  --kv-capacity 32768 \
  --max-concurrency 2 \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft

For the 35B-A3B artifact, select its artifact path; the public model ID follows the container identity automatically:

./build/apps/ninfer-serve models/qwen3_6_35b_a3b.ninfer \
  --max-context 16384 \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft

When --model-id is omitted, the server advertises and accepts the loaded container's exact identity.model_id. An explicit --model-id remains a public HTTP alias override and does not select or alter the artifact.

Vision is disabled by default: its weights, Vision scratch phase, and frozen request-transient buffer are not allocated, and media requests and token-count requests fail with HTTP 400 vision_disabled. Add --vision when the server must accept image or video input. Speculative residency is likewise frozen by --spec mtp|dflash and --draft-tokens; omitting --spec loads neither backend. --lm-head-draft additionally loads the optimized proposal head. DFlash is 35B-A3B text-only and cannot be combined with --vision. A later request cannot enable a capability omitted at startup.

Endpoints

Method and path Behavior
GET /health process health
GET /v1/models configured OpenAI model alias
GET /v1/models/{id} lookup of the configured alias
POST /v1/chat/completions OpenAI-style chat generation
POST /v1/responses OpenAI Responses Core generation, state, typed Items, and SSE
POST /v1/responses/input_tokens Responses prompt-token count without generation
GET /v1/responses/{id} retrieve a locally stored terminal Response
DELETE /v1/responses/{id} delete a locally stored Response
GET /v1/responses/{id}/input_items list that Response's normalized input Items
POST /v1/messages Anthropic-style message generation
POST /v1/messages/count_tokens checkpoint-native expanded input-token count

OpenAI Chat Completions

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.6-27b",
    "messages": [
      {"role": "system", "content": "Answer concisely."},
      {"role": "user", "content": "What is speculative decoding?"}
    ],
    "max_tokens": 128
  }'

The endpoint supports:

  • system, developer, user, assistant, and tool history;
  • string content and ordered text, image_url, and video_url parts;
  • max_completion_tokens and the legacy max_tokens spelling;
  • temperature, top_p, top_k, presence/frequency penalties, and a nonnegative seed;
  • one stop string or an array of stop strings;
  • non-streaming responses and server-sent event streams;
  • stream_options.include_usage;
  • function tools, tool choices, assistant tool-call history, and tool-result messages;
  • the top-level reasoning_effort field;
  • the enable_thinking extension;
  • chat_template_kwargs.preserve_thinking and the top-level preserve_thinking alias.

The request model must equal the public model ID: the artifact identity.model_id by default, or the explicit --model-id override. Reasoning is returned separately as reasoning_content; answer text remains in content.

Message roles retain their input order through schema translation. The Qwen family frontend maps both system and developer to system-class ChatML blocks at their original positions; it does not move later instructions to the beginning of the conversation. A leading instruction keeps the artifact template's existing tool/reasoning-instruction composition.

At startup, NInfer resolves prompt capabilities from the exact frontend/chat_template.jinja resource embedded in the loaded artifact. It does not infer them from the request's model field, the artifact identity, or a target profile. A recognized effort-capable template exposes low, medium, and xhigh; omitting effort uses that template's declared default. An explicit effort not exposed by the loaded template returns HTTP 400 with code reasoning_effort_not_supported before prompt preparation.

For Chat Completions, reasoning_effort: "none" disables thinking. low, medium, and xhigh select the corresponding template effort when available. The other OpenAI protocol values minimal, high, and max are parsed but rejected when the loaded template does not expose them. enable_thinking controls the same new-turn thinking switch; a contradictory combination with reasoning_effort returns conflicting_template_option.

preserve_thinking controls whether reasoning from closed assistant turns remains in later prompts. It defaults to the server setting, which is off unless --preserve-thinking is used. If both OpenAI spellings are present they must carry the same boolean value. Unknown non-null chat_template_kwargs are rejected.

Streaming begins with an assistant-role chunk, sends separate reasoning and content deltas, then a finish-reason chunk and [DONE]. When stream_options.include_usage is true, a final empty choices chunk contains completed usage.

Usage reports the exact resident prompt prefix reused by the Engine as prompt_tokens_details.cached_tokens; prompt_tokens still counts the full prepared prompt:

{
  "usage": {
    "prompt_tokens": 42,
    "prompt_tokens_details": {"cached_tokens": 17},
    "completion_tokens": 12,
    "total_tokens": 54
  }
}

Multimodal request

Start the server with --vision before sending media:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.6-27b",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "image_url", "image_url": {"url": "https://example.com/image.png"}},
        {"type": "text", "text": "Describe this image."}
      ]
    }],
    "max_tokens": 128
  }'

OpenAI image and video sources may be HTTP(S) URLs or base64 data URLs.

Text and media requests use one complete-prompt context contract. After chat-template rendering and media-token expansion, the result must fit Engine --max-context. The current Vision runtime also has a 32,768 merged-token envelope (131,072 raw patches); the effective Vision limit is therefore min(--max-context, 32768). There is no fixed image/video item-count limit: item count is admitted through aggregate source-byte, decoded-pixel, raw-patch, Vision-token, and live-memory budgets.

Media cache misses run as independent decode → resize → BF16-pack tasks on a bounded host worker pool. Prepared payloads are keyed by SHA-256 of the acquired bytes plus modality, so repeated media in later requests reuses the exact immutable BF16 patch input; concurrent identical misses use one single-flight build. --media-cache-mib bounds LRU-retained payloads, while --media-live-mib bounds every cache-, request-, or runtime-referenced payload. Cache eviction does not invalidate a request reference, and live bytes are returned only when the final reference is released. A request-level preparation gate derived from the live limit prevents concurrent partial builds from deadlocking the memory account.

An expanded prompt beyond --max-context returns HTTP 400 context_length_exceeded, including the prepared token count and configured context ceiling. A media preprocessing resource rejection returns HTTP 400 media_budget_exceeded. HTTP 413 request_too_large is reserved for a raw request body that exceeds --max-request-mib before JSON parsing; it is not used for model-context or media resource errors.

OpenAI Responses Core

NInfer implements the typed-Item and semantic-event core of the OpenAI Responses API. All registered artifact identities use this same adapter and Engine route. It is intentionally not advertised as full parity with OpenAI-hosted tools, durable cloud storage, background jobs, Conversations, or compaction.

Create a Response

curl http://127.0.0.1:8080/v1/responses \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.6-27b",
    "instructions": "Answer concisely.",
    "input": "What is speculative decoding?",
    "max_output_tokens": 128,
    "store": true
  }'

The same endpoint works with OpenAI SDKs by replacing their base URL:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="local-secret")
response = client.responses.create(
    model="qwen3.6-27b",
    instructions="Answer concisely.",
    input="What is speculative decoding?",
    max_output_tokens=128,
)
print(response.output_text)  # SDK helper derived from response.output

output_text is an SDK convenience property. It is not emitted as a top-level wire field; the wire response contains typed output Items.

Create request fields

Field NInfer Responses Core contract
model required non-empty string; must equal the artifact-derived public model ID or explicit --model-id override
input required string or non-empty typed Item array
instructions optional string, inserted before the reconstructed conversation for this request only
previous_response_id optional ID of a retained local Response
max_output_tokens integer at least 16; default is --default-max-tokens
stream boolean; true selects Responses SSE rather than a JSON body
store boolean, default true; controls local retrieval and continuation state
temperature finite number in [0,2]
top_p finite number in [0,1]
metadata at most 16 string pairs; keys at most 64 characters and values at most 512
reasoning.effort none disables thinking; low, medium, or xhigh selects an effort exposed by the loaded chat template; minimal, high, and max return reasoning_effort_not_supported for the registered templates
chat_template_kwargs.preserve_thinking optional boolean controlling whether closed-turn reasoning remains in reconstructed prompts
preserve_thinking top-level alias for the same option; conflicting values are rejected
text.format omitted or {"type":"text"} only
tools flat Responses function definitions; see below
tool_choice auto or none
parallel_tool_calls omitted or true
truncation omitted or disabled; overlong input fails instead of silently dropping Items
top_logprobs omitted or 0
service_tier omitted, auto, or default; the response reports default
background omitted or false
include omitted or an empty array
stream_options omitted or {"include_obfuscation":false}

Unknown top-level fields fail with unknown_parameter. Recognized but unsupported features fail with a field-specific 400 error instead of being silently ignored.

Input Item contract

String input is normalized to one user message with an input_text part. Array input accepts:

Item Supported form
message roles user, assistant, system, and developer; string content or typed content array
input_text message content part containing string text
output_text assistant-message replay part containing string text
input_image user-message part with HTTP(S) or data-URI image_url; detail omitted or auto; requires server --vision
input_video NInfer extension with HTTP(S) or data-URI video_url; requires server --vision
reasoning raw replay Item with an empty summary and reasoning_text content parts
function_call completed assistant call with optional id, and required call_id, name, and JSON-object string arguments
function_call_output completed tool result with required call_id and string output

Adjacent function-call Items are grouped into one assistant history turn. A reasoning Item attaches to the following assistant message or function call. Input Item IDs are preserved when supplied and generated otherwise; duplicate IDs fail.

System and developer message Items retain their positions in the input array. Top-level instructions is represented as a leading developer turn for the current request; target-specific role lowering occurs only in the Qwen family frontend.

input_file, input_audio, image file_id, non-auto image detail, reasoning summaries or encrypted reasoning, message phase, and other Item/content types are not supported. HTTP media URLs stored in a response chain are fetched again when that chain is continued; use data URIs when the historical media bytes must be immutable.

Function tools

Responses function definitions are flat rather than Chat Completions' nested function object:

{
  "type": "function",
  "name": "get_weather",
  "description": "Get current weather",
  "parameters": {
    "type": "object",
    "properties": {"city": {"type": "string"}},
    "required": ["city"]
  },
  "strict": false
}

NInfer renders these definitions in the Qwen prompt and parses model output into separate function_call output Items. Each output has a protocol Item id (fc_...) and a distinct call_id (call_...). The client executes the function and sends a function_call_output Item in a later request. NInfer does not execute functions or enforce JSON Schema through constrained decoding, so strict:true, tool_choice:required, named tool choice, hosted tools, MCP tools, and custom free-form tools are rejected.

Response object and usage

A terminal wire response has object: "response", one of completed, incomplete, or cancelled in status, and a typed output array. NInfer may emit:

  • a reasoning Item containing raw reasoning_text and an empty summary;
  • an assistant message containing an output_text part;
  • one or more function_call Items.

Ordinary model/string stops produce completed. Output-token or context-capacity exhaustion produces incomplete with incomplete_details.reason: "max_output_tokens". Errors accepted after an SSE response has started produce response.failed; validation and preparation errors remain normal HTTP error responses.

Usage is checkpoint-native:

{
  "input_tokens": 42,
  "input_tokens_details": {"cached_tokens": 17},
  "output_tokens": 12,
  "output_tokens_details": {"reasoning_tokens": 5},
  "total_tokens": 54
}

input_tokens includes the chat template and expanded media tokens. cached_tokens is the exact resident prompt prefix reused by Engine. output_tokens is the count of accepted generated token IDs, including a withheld stop token when applicable. reasoning_tokens is counted in the Qwen output decoder while accepted tokens are still in the reasoning channel; it is not estimated by re-tokenizing decoded text.

Responses streaming

Set stream:true for semantic Server-Sent Events. Every frame uses both the SSE event name and a matching JSON type, and every JSON event has a monotonically increasing sequence_number:

event: response.output_text.delta
data: {"type":"response.output_text.delta","sequence_number":7,...}

The normal lifecycle is:

  1. response.created, then response.in_progress;
  2. response.output_item.added and response.content_part.added;
  3. zero or more response.reasoning_text.delta or response.output_text.delta events;
  4. matching *.done, response.content_part.done, and response.output_item.done events;
  5. exactly one response.completed, response.incomplete, or response.failed terminal event.

Function arguments use response.function_call_arguments.delta and .done. IDs, output indices, and content indices remain stable, and concatenated deltas equal the terminal Item. Responses SSE does not emit the Chat Completions [DONE] sentinel. With tools enabled, ordinary answer text still streams immediately; only an ambiguous <tool_call> suffix or the structured tool region is held. Malformed tool markup is flushed back as ordinary text without losing bytes.

Local response state and resources

store defaults to true. Stored Responses live only in this server process and are bounded by an LRU store. They are lost on restart and are not OpenAI's durable cloud retention service.

previous_response_id reconstructs the complete stored input/output Item history before the new input. The current instructions value is placed first but is not saved into the continuation context, matching the Responses rule that previous top-level instructions do not carry forward. Function definitions are request configuration rather than conversation Items and must be sent again on tool-result turns. The reconstructed prompt follows the ordinary Engine path, so resident prefix reuse applies naturally.

A stored Response also retains its resolved preserve_thinking value. A child which omits the field inherits the parent value. An explicit different value creates a new semantic branch; prompt rendering and identity still determine reuse. Changing the boolean alone never invalidates an exact current frontier or a complete matching rewrite checkpoint.

Resource behavior:

Endpoint Contract
GET /v1/responses/{id} returns the stored terminal object, or 404 response_not_found
DELETE /v1/responses/{id} removes public retrieval and returns response.deleted; descendant contexts already retained by other Responses remain usable
GET /v1/responses/{id}/input_items returns normalized Items supplied to that request; supports after, limit 1..100 (default 20), and order `asc
POST /v1/responses/{id}/cancel explicitly fails because background execution is unsupported
POST /v1/responses/compact explicitly fails with compaction_not_supported

store:false Responses cannot be retrieved or used as previous_response_id. LRU eviction and explicit deletion also make an ID unavailable. A single Response larger than the configured store capacity fails with response_store_capacity_exceeded rather than silently pretending it was stored.

Responses input token count

POST /v1/responses/input_tokens accepts exactly model and input, performs the same typed Item, template, and media expansion, and does not run generation:

curl http://127.0.0.1:8080/v1/responses/input_tokens \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.6-27b","input":"Count this prompt."}'
{"object":"response.input_tokens","input_tokens":11}

Unsupported Create fields include Conversations, prompt templates, context management, hosted moderation, prompt-cache controls, safety/user identifiers, Structured Outputs/JSON mode, non-empty include, background execution, compaction, files/audio, and OpenAI-hosted/MCP/custom tools. These are compatibility boundaries, not silently accepted placeholders.

Anthropic Messages

curl http://127.0.0.1:8080/v1/messages \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.6-27b",
    "max_tokens": 128,
    "messages": [
      {"role": "user", "content": "Explain prefix reuse in one sentence."}
    ]
  }'

The endpoint supports top-level system text, ordered mid-conversation system messages, user/assistant history, text and image blocks, thinking blocks, tool-use history, tool results, client-defined tools, non-streaming responses, and Anthropic SSE events. Mid-conversation system messages remain at their messages array position and are not merged into the top-level system instruction. A system section must follow a user/tool-result message and be final or immediately precede an assistant message; it cannot interrupt a tool-use/tool-result pair. Consecutive system messages remain separate ordered turns.

thinking.type: "disabled" disables thinking; other supported values enable it. The independent top-level preserve_thinking boolean controls closed-turn history and otherwise uses the server default.

Anthropic output_config.effort accepts the protocol values low, medium, high, xhigh, and max. The value is then checked against the loaded chat template in the same way as the OpenAI endpoints; the registered effort-capable template exposes low, medium, and xhigh. Combining an effort with thinking.type: "disabled" is rejected as contradictory.

Anthropic's model field is treated as a response label and does not select the loaded artifact.

POST /v1/messages/count_tokens uses the artifact's tokenizer, chat template, and media expansion without running GPU generation:

curl http://127.0.0.1:8080/v1/messages/count_tokens \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.6-27b",
    "messages": [{"role": "user", "content": "Count this prompt."}]
  }'

Authentication and CORS

Pass --api-key VALUE to require the same value as an OpenAI bearer token or Anthropic x-api-key header. GET /health and CORS preflight requests remain unauthenticated.

curl http://127.0.0.1:8080/v1/models \
  -H 'Authorization: Bearer local-secret'

--cors adds permissive browser CORS headers. It is disabled by default.

Server options

Option Meaning Default
--host H listen address 127.0.0.1
--port N listen port 8080
--api-key KEY required bearer or x-api-key value unset
--model-id ID override the public OpenAI model alias artifact identity.model_id
--max-context N logical context ceiling of each sequence 8192
--kv-capacity N|auto explicit shared Main Text KV capacity, or maximize it from remaining GPU memory; omitted means --max-context 8192
--max-concurrency N maximum admitted requests; valid range 1..8 1
--max-pending-requests N additional requests allowed to wait for admission 16
--pending-timeout-ms N maximum preparation-plus-admission wait 30000
--prefill-chunk N text-prefill chunk 1024
--log-stats-interval-ms N aggregate throughput report interval; 0 disables it 5000
--device N CUDA device index 0
--max-request-mib N body-size limit before JSON parsing 384
--media-cache-mib N LRU-retained prepared BF16 media payloads; 0 disables retention 1024
--media-live-mib N all live prepared BF16 media payloads 2048
--media-preprocess-threads N bounded media preprocessing workers; 0 selects at most 16 from host concurrency 0
--request-log-jsonl FILE append full-precision server/request records disabled
--response-store-max-records N maximum locally retained Responses objects 1024
--response-store-max-mib N total local Response envelope/Item/context budget 256
--kv-dtype bf16|int8 KV-cache storage bf16
--spec mtp|dflash speculative backend off
--draft-tokens N MTP 1..5; DFlash 1..15 unset
--lm-head-draft optimized proposal head off
--default-max-tokens N output limit when omitted by a request 8192
--vision enable media input and load Vision GPU allocations off
--no-cuda-graph disable CUDA Graph decode graphs on
--no-prefix-reuse disable compatible-prefix caching prefix reuse on
--cold-tier-device N pin the cold tier to one secondary device; -1 spans every non-primary device with at least 512 MiB free -1
--cold-tier-capacity-mib N per-device cold-tier arena bytes; 0 sizes each arena from device free memory minus 256 MiB 0
--cold-tier-staging-mib N pinned host staging buffer for cold-tier transfers 64
--no-cold-tier disable the cross-GPU cold tier cold tier on
--no-thinking disable thinking by default thinking on
--preserve-thinking preserve closed-turn assistant reasoning by default off
--cors permissive browser CORS headers off
--temperature F process-level temperature override unset
--top-p F process-level top-p override unset
--top-k N process-level top-k override unset
--min-p F process-level min-p override unset
--presence-penalty F process-level presence-penalty override unset
--frequency-penalty F process-level frequency-penalty override unset
--seed N fixed seed when a request omits one fresh random seed per request
--greedy force exact argmax for all requests off

Engine selects sampling defaults from the loaded model and the request's resolved thinking mode. Qwen3.6-27B and Qwen3.8-27B use 1.0/0.95/20/0/0 for temperature/top-p/top-k/min-p/presence penalty in thinking mode and 0.7/0.80/20/0/1.5 in non-thinking mode. Qwen3.6-35B-A3B differs only in its thinking presence penalty, which is 1.5. Frequency penalty is 0 for all registered presets. Process flags override registered values, request fields override process flags, and --greedy finally forces temperature 0.

Run ./build/apps/ninfer-serve --help for the exact option contract.

Cross-GPU cold tier

With more than one GPU present, the server keeps evicted retained prefixes in a VRAM cold tier on the secondary devices instead of dropping them. A retained resident is parked into the tier whenever its lane is taken from it: either by an admission eviction, or by an in-place full reset on the same lane when a different conversation starts there. Its serialized state image (paged KV pages, MTP backend KV, GDN slots, tail and rewrite-checkpoint hidden rows) is swapped through pinned host memory into LRU arenas on the secondary devices. When a later request presents the same prefix, the image is swapped back into a freshly materialized allocation before lane selection, so the request resumes through the normal prefix-reuse path instead of re-prefilling. A parked entry matches a prompt at the full ledger when the prompt re-echoes the parked reply (the request appends at the execution frontier), or at the turn checkpoint when the next request re-renders the final turn — the generation prompt and thinking control diverge the parked ledger's tail — in which case the planner restores the checkpoint and re-prefills only the re-rendered tail, exactly as on a resident lane. Under pool pressure the admission pass parks other slot-free retained residents into the tier first, so multiple concurrent conversations swap state rather than evicting each other to a full reset.

The tier is best-effort: a missing secondary device, an oversize image, or a failed transfer falls back to the normal admission path without failing the request. It changes neither the FIFO admission order nor the reuse paths a request can take. The request log records cold_tier_parks, cold_tier_restores, cold_tier_evictions, cold_tier_park_failures, cold_tier_restore_failures, cold_tier_capacity_bytes, and arena occupancy. Each request-done stderr line appends the cumulative cold=parks=… restores=… failures=… entries=… counters while the tier is active — including evictions=… (LRU entries dropped for capacity) and used=…GiB (current arena occupancy) — and startup logs the arena count, total capacity, and staging buffer size. A growing evictions with no matching restores means the workload holds more parked frontiers than the arenas can retain, so the least-recently-used conversations age out before they return.

Structured request log

--request-log-jsonl FILE enables the machine-readable measurement log. The server opens FILE in append mode and flushes every event, so successive model or MTP blocks may share one campaign file. The parent directory must already exist. Failure to open the file aborts startup; the log path is also rejected if it resolves to the model artifact.

./build/apps/ninfer-serve models/qwen3_6_27b.ninfer \
  --request-log-jsonl profiles/bench/run/server.requests.jsonl

Every line is one ninfer_serve_request_log schema-v12 JSON object. All events carry timestamp_unix_ms and a process-unique server_instance_id; request IDs are monotonic only within that server instance. Successful request-start records include request-scoped acquisition, media-preprocessing wall/work, tokenizer, cache hit/miss/single-flight, and payload-size fields; they do not infer request behavior from process-global counter deltas.

Event Contents
server_start target/weights identity and artifact, resolved Engine, registered thinking/non-thinking sampler defaults plus process overrides, thinking-history defaults, weights/sequence/workspace/request-transient arenas, KV sizing ledger, CUDA Graph observed/allowance bytes, CUDA/GPU environment, and redacted argv
request_start protocol, resolved sampler and seed, thinking modes, Responses semantic-change flag, output budget, stream/message/tool shape
request_rejected parsed request shape, media-item count, phase: "prepare", and the exact HTTP status/type/code/parameter/message for a synchronous preparation rejection
request_done finish reason, prompt/completion/cache/computed-prefill tokens, prefix reuse path, unrounded phase seconds, and complete speculative-decoding counters
request_error the resolved request configuration and generation error message
throughput interval token deltas and rates, scheduler occupancy, and decode-round batch statistics

request_done.timings_seconds contains prepare, ttft, vision, prefill, decode, and total as full-precision JSON numbers. Its speculative object contains backend, draft_window, rounds, drafted_tokens, accepted_tokens, fallback_steps, and accepted_per_position. Rates can be derived downstream from raw token counts and seconds instead of rounded stderr strings.

The JSONL file contains no generated response text and never records an API-key value; argv replaces that value with <redacted>. The existing stderr summaries remain available for operators but are rounded and are not the aggregation source. Console lines use local [YYYY-MM-DD HH:MM:SS.mmm] [level] timestamps. OpenAI Responses, OpenAI Chat, and Anthropic generation requests receive a request ID when they enter synchronous preparation. Successful preparation produces request_start; a preparation failure produces request_rejected without a matching start. Later generation failures produce request_error. Schema/model validation rejections before preparation and token-count-only calls are not measurement requests and do not receive request IDs.

By default the server also reports aggregate activity every five seconds. prefill counts prompt suffix tokens actually computed during the interval, excluding prefix-cache hits; decode counts tokens finally committed by decode rounds, excluding the first token produced by prefill. For MTP and DFlash this is the accepted committed output, not draft or rejected tokens. avg_decode_batch is decode row-rounds divided by decode rounds during the same interval. The running, prefilling, decode_ready, and waiting fields are the Engine scheduler snapshot at the end of the interval. Fully idle zero intervals are omitted. The JSONL throughput event keeps the raw token and round deltas as well as derived rates; downstream measurement should prefer those raw values.

Execution behavior

The server owns one resident Engine with a startup-fixed capacity of 1..8 active generation requests. At each decode boundary, every decode-ready request is compacted into one batch and processed by one model traversal and, when graphs are enabled, one exact-batch CUDA Graph replay. A request joins that batch only after its single-request prefill finishes; when it completes or is cancelled, the next boundary rebuilds the batch without an empty row.

--max-pending-requests bounds the requests waiting behind the active set. The total generation request lifetime capacity is max_concurrency + max_pending_requests, including requests still in CPU/media preparation and completed model results whose response has not yet been released. A full capacity returns HTTP 429 with code server_overloaded. The absolute --pending-timeout-ms deadline starts before preparation, covers media acquisition and Engine FIFO waiting, and returns HTTP 503 with code request_queue_timeout if admission does not occur in time. There is no admission ETA or unbounded overflow queue.

Input memory is bounded by the outstanding-request count and the per-request --max-request-mib limit. Media requests additionally share one preparation permit, so a waiting media request retains the same cancellation and timeout deadline. Model output is bounded by the same finite request count and each request's effective output-token limit; output callbacks and network serialization run outside the GPU executor and do not delay formation of the next batch.

--max-context and the resolved --kv-capacity are independent limits. The former is each sequence's logical ceiling; the latter sizes the shared Main Text KV pool used by all active requests and retained prefixes. Both are represented with 64-token pages internally, while a sequence can never cross the exact --max-context frontier. --kv-capacity N requests an explicit capacity; --kv-capacity auto chooses the largest legal capacity that fits the memory remaining after weights are loaded while keeping 1 GiB of sizing headroom. When omitted it follows --max-context, preserving one full-length request's capacity. The shared pool is fixed at startup and is not divided evenly among request lanes.

Automatic sizing evaluates the complete target runtime layout for the chosen concurrency, KV dtype, speculative backend, draft window, Vision setting, workspace, and CUDA Graph allowance. It uses a direct page-capacity calculation rather than allocation probing. Startup reports the policy, resolved capacity, runtime reservation, free memory after weights, automatic headroom, planned slack, actual free memory after complete startup, and observed Graph memory. An explicit capacity is never silently reduced, and neither policy permits request-time pool growth.

Admission reserves the full prompt-plus-effective-output page entitlement, so an admitted request can finish within its declared bound. A later request waits in FIFO order when the remaining shared pages cannot satisfy its complete entitlement; the Engine never admits it and later truncates an older request to recover capacity. Startup rejects a KV pool smaller than one sequence, too small to provide one page per configured lane, or larger than all configured lanes could use.

Compatible resident prefixes are reused for both text and multimodal histories unless the server is started with --no-prefix-reuse. A multimodal hit requires matching token types, three-axis MRoPE positions, encoded-media digest, grid, and consumer spans; changing an earlier image or video therefore resets the prefix instead of reusing placeholder-token KV. Media wholly inside a matched prefix skips Vision execution, while new suffix media is encoded normally. The completion log reports the reused token count as cache=.

The shared family runtime distinguishes full_reset, append_frontier, restore_turn_checkpoint, and restore_response_checkpoint. Both checkpoint kinds include the recurrent, hidden, and selected speculative-backend continuation state required to recompute a rewritten suffix; matching KV tokens alone never authorize a partial hit. With stable preserve_thinking=true, the auxiliary checkpoint rolls to the prompt frontier after the current response's complete deterministic generation prologue. For thinking generation this includes <think>\n; for non-thinking generation it includes the complete empty thinking block. Capturing that frontier does not split a tiny trailing prologue into a separate prefill unit, and a normalized response which no longer matches the raw generated tokens replays only that response and its suffix. Stable false keeps the first assistant opener in the open turn so a newly closed turn can be recomputed without its reasoning.

preserve_thinking selects where the next checkpoint should live; it is not a cache-compatibility bit. An exact current frontier or matching complete checkpoint remains reusable across a mode change. If the newly desired boundary is already behind the selected reuse frontier and no snapshot exists there, the Engine keeps the valid hit and defers installing that new checkpoint rather than forcing an eager full reset. A later request that diverges before every retained checkpoint then resets normally. The JSONL completion record exposes the checkpoint actually restored as prefix_reuse_path. Changing reasoning effort changes rendered tokens and therefore does not reuse a prefix whose effort instruction differs.

An appended mid-conversation system message is an ordinary prompt suffix, so an unchanged prior history remains eligible for append_frontier. If the client modifies, removes, or moves a historical system message, the token prefix genuinely differs and a miss/reset is correct.

Speculative decoding is an engine option and does not change protocol output shapes, stop behavior, or usage accounting. If a stop truncates a multi-token MTP or DFlash round, the Engine commits the exact accepted target prefix so a following compatible turn can still reuse it. Output-limit and context-capacity finishes map to length/ max_tokens; ordinary model or string stops map to stop/ end_turn.

Function tools are rendered into the model prompt and generated calls are parsed into protocol responses. NInfer does not execute tools and does not enforce client JSON Schema through constrained decoding.

Prompt-token usage includes chat-template and expanded media tokens. Generated-token usage comes from accepted output token IDs, including a stop token whose decoded text may be withheld.