Skip to content

[Presets] Make a preset workload state the requested dataset - #4201

Merged
peterschmidt85 merged 1 commit into
masterfrom
preset-workload-kind
Aug 25, 2026
Merged

[Presets] Make a preset workload state the requested dataset#4201
peterschmidt85 merged 1 commit into
masterfrom
preset-workload-kind

Conversation

@peterschmidt85

@peterschmidt85 peterschmidt85 commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Fixes #4198.

random is dstack's name for a synthetic workload; a benchmark tool has its own name for the data
it generates — random in vLLM, generated-shared-prefix in SGLang. Verification compared the
two as one name, so a preset with shared_prefix_tokens and no dataset failed whenever the
agent benchmarked with SGLang, and passed with vLLM only because the names coincide.

Now dataset states the dataset you asked for, and nothing else.

type: preset
base: Qwen/Qwen3.8-27B

input_tokens: 131072
output_tokens: 512
shared_prefix_tokens: 130048

No dataset, so the benchmark runs synthetic prompts. Verification compares the shared prefix and
the concurrency with the request; whatever the tool calls its generated data stays in
benchmark.command, and the stored preset records no dataset.

type: preset
base: Qwen/Qwen3.8-27B

dataset: spec_bench

dataset names a real dataset, which defines its own requests. Verification checks the benchmark
served exactly it, and every mismatch names both values.

The UI shows a preset's dataset only when one was requested.

Breaking changes

  • dataset: random is rejected: omit dataset for synthetic prompts.

Backward compatibility

  • Presets and in-flight sessions that recorded dataset: random upgrade when read.

Reworks #4199 by @r4victor; its verification comparisons and much of its test suite carry over.
Reworked because:

  1. Fix preset verification rejecting a tool's synthetic dataset name #4199 kept the tool's name in workload.dataset, and the pulled-preset row from [Presets] Support for dstack preset push and dstack preset pull #4194 reads
    that field as the request — the same confusion, one layer up.
  2. A field that echoes the request for a named dataset but records a tool detail for a synthetic
    one has two meanings; now it has one.
  3. The contract holds without system prompt changes: the stored workload is normalized from the
    configuration.

AI assistance: written with Claude Code.

`random` was dstack's own name for a synthetic workload, but the report's
`dataset` carried the benchmark tool's name for the data it served, so a
shared-prefix benchmark run with SGLang could never verify (#4198): two
namespaces, compared as one.

Now the workload states the request. `dataset` echoes the requested dataset
and is absent for a synthetic workload, whose requested shared prefix is
compared instead - the one fact the request and the report share. The tool's
own name for its generated data stays where it already was, in `command`.
The stored record is normalized from the configuration, the authority on
what was requested, so nothing the agent volunteers can contradict the
contract - and the system prompt needs no change.

`dataset: random` is retired from the configuration: a set dataset always
means a real one, and synthetic prompts are requested by omitting it.
Records it was legal in are upgraded when read - stored presets, in both
the configuration and the workload, and an in-flight session's saved
configuration.

Based on #4199: the verification comparisons and much of
the test suite are Victor's.

Co-authored-by: Victor Skvortsov <vds003@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@peterschmidt85 peterschmidt85 changed the title [Presets] Rebase of #4199 onto master, for review [Presets] Make a preset workload state the requested dataset Aug 25, 2026
@peterschmidt85
peterschmidt85 marked this pull request as ready for review August 25, 2026 12:24
@peterschmidt85
peterschmidt85 merged commit f2a5b68 into master Aug 25, 2026
27 checks passed
@peterschmidt85
peterschmidt85 deleted the preset-workload-kind branch August 25, 2026 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Preset creation fails with a dataset error when no dataset is set

1 participant