Write preset traces by default and add the dstack-presets skill (#4192)

Every preset session now writes the real-time trace to
~/.dstack/presets/<id>/trace.jsonl, along with the agent prompt and the
final report copy; the --debug flag is removed. A new dstack-presets
skill teaches AI agents to create and manage presets; it is not bundled
into the creation agent.

Co-authored-by: Andrey Cheptsov <andrey.cheptsov@github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Author: 3 people

mkdocs/docs/concepts/presets.md

@@ -122,7 +122,7 @@ Alternatively, pass `--fleet` to `dstack apply`.
 
 === "Base"
 
-    Set `base` to let the creation agent select any compatible variant of the base model, including a different precision, quantization, or trusted fork.
+    Set `base` to let the agent select any compatible variant of the base model, including a different precision, quantization, or trusted fork.
 
     ```yaml
     base: Qwen/Qwen2.5-7B-Instruct
@@ -240,6 +240,24 @@ Submit the run dsv4-flash? [y/n]: y
 
 ## Manage presets
 
+### Watch presets
+
+While a preset is being created, you can watch the progress of its trials and what the agent is doing.
+
+The `dstack preset logs` command shows the progress log: one line per milestone, such as a trial finishing or the final service being verified. Pass `-f` to follow a running creation:
+
+<div class="termy">
+
+```shell
+$ dstack preset logs -f c83375b4
+```
+
+</div>
+
+### Traces
+
+The agent subprocess writes real-time traces to `~/.dstack/presets/<id>/trace.jsonl`: the agent's messages and every tool call with its result. Traces are the main way to analyze a session in depth — see [Protips](#protips).
+
 ### List presets
 
 Use `dstack preset` to list presets:
@@ -290,19 +308,14 @@ $ dstack preset delete c83375b4
 !!! info "Reference"
     For command options and agent settings, see the [`dstack preset` CLI reference](../reference/cli/dstack/preset.md).
 
-## Troubleshooting
+## Protips
 
-To trace the agent's activity, pass `--debug` to `dstack apply`:
-
-<div class="termy">
+Under the hood, presets run an agent as a subprocess, using the local `claude` CLI. This process writes a real-time trace to `~/.dstack/presets/<id>/trace.jsonl`. The subprocess is launched with a built-in harness: how to run trials, submit runs, benchmark, verify presets, and use `dstack`.
 
-```shell
-$ dstack apply -f preset.dstack.yml --debug
-```
-
-</div>
+At the same time, it's recommended to create presets using your own agent — either via a CLI such as Claude Code, or inside your IDE. Your agent helps you design the preset configuration, formulate hypotheses, and — most importantly — analyze the session's traces as well as the trial results (stored under `~/.dstack/presets/<id>/trials/<n>/trial.json`), to decide what the next session can be and what instructions to give it via `prompt`.
 
-The trace is written to `~/.dstack/presets/<id>/trace.jsonl` while the session runs. It contains the agent's messages and every tool call with its result.
+> To help your agent use `dstack` and presets, install the [`dstack`](https://skills.sh/dstackai/dstack/dstack)
+> and [`dstack-presets`](https://skills.sh/dstackai/dstack/dstack-presets) skills with `npx skills add dstackai/dstack`.
 
 ## Limitations
 

mkdocs/docs/reference/cli/dstack/preset.md

@@ -49,9 +49,9 @@ Preset creation uses the existing `claude` login unless
 | `DSTACK_AGENT_CLAUDE_EFFORT` | Claude effort level: `low`, `medium`, `high`, `xhigh`, or `max`. If unset, the `claude` CLI default is used. |
 
 Agent progress is written to `agent.log` under `~/.dstack/presets/<preset-id>/`,
-alongside the effective configuration (`preset.dstack.yml`) and the recorded
-trials (`trials.jsonl`). Pass `--debug` to also save the agent prompt
-(`prompt.md`) and raw trace (`trace.jsonl`).
+alongside the effective configuration (`preset.dstack.yml`), the recorded
+trials, the agent prompt (`prompt.md`), and the real-time trace
+(`trace.jsonl`).
 
 ## dstack preset logs
 

skills/dstack-presets/SKILL.md

@@ -0,0 +1,29 @@
+---
+name: dstack-presets
+description: |
+  Create and manage dstack presets: a toolkit that streamlines model inference optimization with agents, and a portable preset format. Use together with the dstack skill, and only when the user explicitly asks to create a preset or manage existing presets, not for deploying or serving a model.
+---
+
+# dstack Presets
+
+Use `/dstack` for CLI commands, YAML fields, apply behavior, fleets, and other
+dstack syntax. This skill covers creating and managing presets.
+
+## Overview
+
+Presets offer two things: a toolkit that streamlines model inference optimization using agents, and a portable format that deploys the final preset to any cloud, Kubernetes cluster, or bare-metal fleet. A preset holds the serving configuration that produced the result, the benchmark it reached, and the exact hardware it was verified on.
+
+Presets are used for three kinds of work: finding an optimized baseline, optimizing through patching source code, and supporting new hardware.
+
+**When to use this skill:**
+- The user explicitly asks to create a preset, or to optimize model inference via a preset
+- Managing already created presets: watching sessions, listing, exporting, and deleting them via `dstack preset` commands
+
+**When NOT to use this skill:**
+- Deploying or serving a model: use a service instead (see the `dstack` skill)
+
+## How to use presets
+
+Follow the [presets documentation](https://dstack.ai/docs/concepts/presets.md).
+
+[Configuration reference](https://dstack.ai/docs/reference/dstack.yml/preset.md) | [CLI reference](https://dstack.ai/docs/reference/cli/dstack/preset.md)

src/dstack/_internal/cli/models/preset_agent.py

@@ -118,7 +118,6 @@ class PresetSessionState(CoreModel):
     trials_num: Optional[int]
     previous: list[str]
     created_at: datetime
-    debug: bool
     status: PresetSessionStatus
     # None is a detached session.
     owner: Optional[PresetSessionProcess]

src/dstack/_internal/cli/services/configurators/preset.py

@@ -66,7 +66,6 @@ def apply_configuration(
                 configuration=conf,
                 store=store,
                 keep_service=configurator_args.keep_service,
-                debug=configurator_args.debug,
                 user_prompt=user_prompt,
                 allowed_fleets=allowed_fleets,
                 previous=previous,
@@ -131,11 +130,6 @@ def register_creation_args(parser: ArgsParser) -> None:
         metavar="N",
         help="The number of benchmarked trials before the best one is promoted",
     )
-    parser.add_argument(
-        "--debug",
-        action="store_true",
-        help="Save the agent prompt and raw trace",
-    )
     parser.add_argument(
         "--previous",
         action="append",

src/dstack/_internal/cli/services/presets/agent.py

@@ -375,7 +375,7 @@ def _prepare_subprocess_command(command: list[str]) -> list[str]:
     return [comspec, "/d", "/s", "/c", subprocess.list2cmdline(command)]
 
 
-def _write_debug_trace(
+def _write_trace(
     session: PresetSession,
     *,
     stream_name: Literal["stdout", "stderr"],
@@ -523,7 +523,7 @@ async def _read_process_stream(
     redacted_values: Sequence[str],
     session: PresetSession,
 ) -> PresetAgentProcessOutput:
-    # stderr feeds the debug trace and advances the persisted offset, but only
+    # stderr feeds the trace and advances the persisted offset, but only
     # stdout can carry the report.
     parse_result = stream_name == "stdout"
     output = PresetAgentProcessOutput()
@@ -532,13 +532,12 @@ async def _read_process_stream(
         if not line:
             return output
         text = line.decode(errors="replace")
-        if session.debug:
-            _write_debug_trace(
-                session,
-                stream_name=stream_name,
-                text=text,
-                redacted_values=redacted_values,
-            )
+        _write_trace(
+            session,
+            stream_name=stream_name,
+            text=text,
+            redacted_values=redacted_values,
+        )
         if not parse_result:
             continue
         try:

src/dstack/_internal/cli/services/presets/create.py

@@ -400,7 +400,6 @@ def create_preset(
     store: PresetStore,
     keep_service: bool = False,
     build_name: Optional[str] = None,
-    debug: bool = False,
     resume_session: Optional[PresetSession] = None,
     user_prompt: Optional[str] = None,
     allowed_fleets: Optional[tuple[str, ...]] = None,
@@ -409,7 +408,6 @@ def create_preset(
     session = resume_session or create_preset_session(
         configuration,
         previous=tuple(session.preset_id for session in previous),
-        debug=debug,
     )
     try:
         resolved_configuration = _resolve_preset_env(configuration)
@@ -629,10 +627,9 @@ async def _create_preset(
         # A second, persistent copy: the workspace above is deleted with the run,
         # while the listing and `--previous` read constraints from the session dir.
         session.write_constraints(constraints_text)
-        if session.debug:
-            session.write_prompt(prompt)
-            if setup.auth is not None:
-                session.write_agent_info(setup.auth)
+        session.write_prompt(prompt)
+        if setup.auth is not None:
+            session.write_agent_info(setup.auth)
     try:
         if mode == "attach":
             process_output = await attach_preset_agent(
@@ -685,12 +682,11 @@ async def _create_preset(
         interrupted = True
         raise
     finally:
-        if session.debug:
-            _save_final_report_copy(
-                workspace=setup.workspace,
-                session=session,
-                redacted_values=redacted_values,
-            )
+        _save_final_report_copy(
+            workspace=setup.workspace,
+            session=session,
+            redacted_values=redacted_values,
+        )
         if not interrupted:
             keep_final_service = keep_service and creation_succeeded
             try:
@@ -1009,8 +1005,7 @@ async def _cleanup_runs(
                     pending.remove(name)
             if pending:
                 await asyncio.sleep(2)
-    if session.debug:
-        print_preset_progress("All preset creation runs stopped.", session=session)
+    print_preset_progress("All preset creation runs stopped.", session=session)
 
 
 def _load_submitted_run_names(path: Path) -> list[str]:

src/dstack/_internal/cli/services/presets/session.py

@@ -56,7 +56,6 @@ class SessionBusyError(CLIError):
 @dataclass
 class PresetSession:
     path: Path
-    debug: bool
     preset_id: str
     # Background reconcile sets this False so finalizing a detached session stays
     # silent on the read command; agent.log is written regardless.
@@ -269,7 +268,6 @@ def create_preset_session(
     configuration: PresetConfiguration,
     *,
     previous: Sequence[str],
-    debug: bool,
 ) -> PresetSession:
     if configuration.name is None:
         raise CLIError("The service name is required to save agent output")
@@ -286,7 +284,8 @@ def create_preset_session(
                 continue
             break
         _write_private_text(path / "agent.log", "")
-        session = PresetSession(path=path, debug=debug, preset_id=preset_id)
+        _write_private_text(path / "trace.jsonl", "")
+        session = PresetSession(path=path, preset_id=preset_id)
         session.write_state(
             PresetSessionState(
                 id=preset_id,
@@ -295,7 +294,6 @@ def create_preset_session(
                 trials_num=configuration.trials,
                 previous=list(previous),
                 created_at=datetime.now(timezone.utc),
-                debug=debug,
                 status="running",
                 owner=_current_process(),
                 run=None,
@@ -311,8 +309,6 @@ def create_preset_session(
             path / "preset.dstack.yml",
             yaml.safe_dump(record, sort_keys=False),
         )
-        if debug:
-            _write_private_text(path / "trace.jsonl", "")
     except OSError as e:
         if path is not None:
             shutil.rmtree(path, ignore_errors=True)
@@ -322,7 +318,7 @@ def create_preset_session(
 
 def load_resumable_session(preset_id: str) -> PresetSession:
     path = get_presets_dir() / preset_id
-    session = PresetSession(path=path, debug=False, preset_id=preset_id)
+    session = PresetSession(path=path, preset_id=preset_id)
     state = session.read_state()
     if not path.is_dir() or state is None:
         raise CLIError(f"Unknown preset: {preset_id}")
@@ -337,7 +333,6 @@ def load_resumable_session(preset_id: str) -> PresetSession:
         )
     if state.run is None or state.run.claude_session_id is None:
         raise CLIError(f"Preset {preset_id} creation stopped before it started; create a new one")
-    session.debug = state.debug
     return session
 
 
@@ -368,7 +363,7 @@ def session_process_alive(state: PresetSessionState) -> bool:
 
 def load_attachable_session(preset_id: str) -> PresetSession:
     path = get_presets_dir() / preset_id
-    session = PresetSession(path=path, debug=False, preset_id=preset_id)
+    session = PresetSession(path=path, preset_id=preset_id)
     state = session.read_state()
     if not path.is_dir() or state is None:
         raise CLIError(f"Unknown preset: {preset_id}")
@@ -387,13 +382,12 @@ def load_attachable_session(preset_id: str) -> PresetSession:
             f"Preset {preset_id} is already being followed by another CLI (pid {owner.pid});"
             f" stop or detach it there with Ctrl+C"
         )
-    session.debug = state.debug
     return session
 
 
 def load_preset_session(preset_id: str) -> PresetSession:
     path = get_presets_dir() / preset_id
-    session = PresetSession(path=path, debug=False, preset_id=preset_id)
+    session = PresetSession(path=path, preset_id=preset_id)
     if not path.is_dir() or session.read_state() is None:
         raise CLIError(f"Unknown preset: {preset_id}")
     return session
@@ -476,7 +470,7 @@ def iter_preset_sessions() -> Iterator[PresetSession]:
         return
     for path in sorted(root.iterdir()):
         if path.is_dir() and not path.name.startswith((".", "models--")):
-            yield PresetSession(path=path, debug=False, preset_id=path.name)
+            yield PresetSession(path=path, preset_id=path.name)
 
 
 def find_session_name_claims(name: str) -> list[PresetSession]:

src/tests/_internal/cli/commands/test_preset.py

@@ -485,7 +485,6 @@ def test_merges_profile_configuration_and_cli_args(self, tmp_path):
                     "0.75",
                     "--fleet",
                     "cli-fleet",
-                    "--debug",
                 ],
                 home_dir=tmp_path,
                 repo_dir=tmp_path,
@@ -499,7 +498,6 @@ def test_merges_profile_configuration_and_cli_args(self, tmp_path):
         assert configuration.max_price == 0.75
         assert configuration.spot_policy.value == "spot"
         assert [fleet.format() for fleet in configuration.fleets] == ["cli-fleet"]
-        assert create.call_args.kwargs["debug"] is True
 
     def test_create_detaches_the_name_from_the_old_preset(self, tmp_path):
         preset = get_preset().model_copy(update={"name": "qwen"})
@@ -630,7 +628,6 @@ def test_accepts_creation_and_profile_arguments(self, tmp_path):
                     "7",
                     "--backend",
                     "gcp",
-                    "--debug",
                 ],
                 home_dir=tmp_path,
                 repo_dir=tmp_path,
@@ -641,7 +638,6 @@ def test_accepts_creation_and_profile_arguments(self, tmp_path):
         assert configuration.name == "cli-name"
         assert configuration.trials == 7
         assert configuration.backends == ["gcp"]
-        assert create.call_args.kwargs["debug"] is True
 
     def test_rejects_detach(self, tmp_path, capsys):
         configuration_path = self._write_configuration(tmp_path)

src/tests/_internal/cli/common.py

@@ -210,7 +210,6 @@ def get_session_state(**overrides: Any) -> PresetSessionState:
         "trials_num": None,
         "previous": [],
         "created_at": datetime(2026, 1, 2, 3, 4, tzinfo=timezone.utc),
-        "debug": False,
         "status": "running",
         "owner": None,
         "run": None,