[Bug]: A rolling deployment whose new replica cannot start terminates the healthy service

[r4victor]

Steps to reproduce

1. Create a fleet that cannot grow beyond one node:

type: fleet
name: rolling-repro
nodes: 0..1
resources:
  cpu: 2..

2. Deploy a single-replica service onto it:

type: service
name: rolling-repro-svc
image: python:3.12-slim
port: 8000
auth: false
commands:
  - mkdir -p /srv && cd /srv
  - echo v1 > index.html
  - python -m http.server 8000
resources:
  cpu: 2..

3. Wait until the service is running and serving.
4. Change commands (echo v2 > index.html) and dstack apply again, accepting the in-place update.

Actual behaviour

Rather than leaving the running version alone, dstack terminates the run and the healthy old replica with it, so the service goes down permanently.

Expected behaviour

A rolling deployment whose new replica cannot start should leave the previous deployment serving and report that the rollout failed. Destroying a healthy, known-good deployment because its replacement could not be placed is worse than the update not happening at all.

dstack version

master 1298b421a (source checkout)

Server logs

Additional information

Where it happens. The run-level failure analysis is deployment-blind. Any replica with a failed job and no retry contributes FAILED + JOB_FAILED:

dstack/src/dstack/_internal/server/background/pipeline_tasks/runs/active.py

Lines 220 to 229 in 1298b42

	if _job_needs_retry_evaluation(job_model):
	current_duration = await _should_retry_job(run_model, job_model)
	if current_duration is None:
	contributed_statuses.add(RunStatus.FAILED)
	termination_reasons.add(RunTerminationReason.JOB_FAILED)
	elif _is_retry_duration_exceeded(job_model, current_duration):
	contributed_statuses.add(RunStatus.FAILED)
	termination_reasons.add(RunTerminationReason.RETRY_LIMIT_EXCEEDED)
	else:
	needs_retry = True

and one such replica then decides the whole run, discarding what every other replica contributed:

dstack/src/dstack/_internal/server/background/pipeline_tasks/runs/active.py

Lines 247 to 250 in 1298b42

	if RunStatus.FAILED in replica_analysis.contributed_statuses:
	analysis.contributed_statuses.add(RunStatus.FAILED)
	analysis.termination_reasons.update(replica_analysis.termination_reasons)
	return

Neither reads deployment_num, even though the rollout logic next door is built entirely on it (build_replica_lists, get_group_rollout_state, has_out_of_date_replicas in services/runs/replicas.py). There is also no rollback or "abandon the failed deployment" concept anywhere in the run/service services.

How it got here: #986 introduced "any failed replica fails the whole run", in a form structurally identical to today's code:

dstack/src/dstack/_internal/server/background/tasks/process_runs.py

Lines 206 to 216 in 763bc16

	else:
	# just failed
	replica_statuses.add(RunStatus.FAILED)
	run_termination_reasons.add(RunTerminationReason.JOB_FAILED)
	elif job.status in {JobStatus.TERMINATING, JobStatus.TERMINATED, JobStatus.ABORTED}:
	pass # unexpected, but let's ignore it
	else:
	raise ValueError(f"Unexpected job status {job.status}")
	if RunStatus.FAILED in replica_statuses:
	run_statuses.add(RunStatus.FAILED)

#2821 then added deployment_num and the in-place update path on top. It changed process_runs.py substantially, but not the failure/termination analysis — the only deployment_num lines it adds there concern bumping and out-of-date detection.