Steps to reproduce
- Create a fleet with blocks that is allowed to grow:
type: fleet
name: blocks-grow
nodes: 0..2
blocks: auto
resources:
cpu: 4..8
memory: 8GB..32GB
- Fill every block on the first instance by submitting one task per block:
type: task
name: filler
image: python:3.12-slim
commands:
- sleep 3600
resources:
cpu: 1
memory: 2GB..
- Submit one more task with the same
resources (cpu: 1).
- Submit another task with
resources: cpu: 4..8, memory: 8GB..32GB.
Actual behaviour
The task in step 3 fails immediately, although the fleet is still below nodes.max and could add a node:
Job status changed SUBMITTED -> TERMINATING. Termination reason: FAILED_TO_START_DUE_TO_NO_CAPACITY
termination_reason_message is None, so nothing indicates why. The task in step 4 provisions a second instance within ~20s against the same fleet, so the fleet itself is able to grow.
Expected behaviour
The fleet provisions a new instance and packs the job into a block, the same way jobs are packed onto blocks of an instance that already exists.
dstack version
master 1298b421a (source checkout)
Server logs
job filler-0-0: provisioning has started
job filler-0-0: Cannot combine fleet requirements
job filler-0-0: provisioning failed
Emitting event: Job status changed SUBMITTED -> TERMINATING. Termination reason: FAILED_TO_START_DUE_TO_NO_CAPACITY
Additional information
The run's requirements are intersected with the fleet's per-instance requirements when provisioning new capacity. A bare cpu: 1 parses as exactly 1..1, which does not intersect the fleet's 4..8, so the fleet yields no offers. Placement on an existing instance is unaffected because those offers are derived per block.
Steps to reproduce
resources(cpu: 1).resources: cpu: 4..8, memory: 8GB..32GB.Actual behaviour
The task in step 3 fails immediately, although the fleet is still below
nodes.maxand could add a node:termination_reason_messageisNone, so nothing indicates why. The task in step 4 provisions a second instance within ~20s against the same fleet, so the fleet itself is able to grow.Expected behaviour
The fleet provisions a new instance and packs the job into a block, the same way jobs are packed onto blocks of an instance that already exists.
dstack version
master
1298b421a(source checkout)Server logs
Additional information
The run's requirements are intersected with the fleet's per-instance requirements when provisioning new capacity. A bare
cpu: 1parses as exactly1..1, which does not intersect the fleet's4..8, so the fleet yields no offers. Placement on an existing instance is unaffected because those offers are derived per block.