Skip to content

[Bug]: A fleet with blocks cannot provision a new instance for jobs smaller than one node #4188

Description

@r4victor

Steps to reproduce

  1. Create a fleet with blocks that is allowed to grow:
type: fleet
name: blocks-grow
nodes: 0..2
blocks: auto
resources:
  cpu: 4..8
  memory: 8GB..32GB
  1. Fill every block on the first instance by submitting one task per block:
type: task
name: filler
image: python:3.12-slim
commands:
  - sleep 3600
resources:
  cpu: 1
  memory: 2GB..
  1. Submit one more task with the same resources (cpu: 1).
  2. Submit another task with resources: cpu: 4..8, memory: 8GB..32GB.

Actual behaviour

The task in step 3 fails immediately, although the fleet is still below nodes.max and could add a node:

Job status changed SUBMITTED -> TERMINATING. Termination reason: FAILED_TO_START_DUE_TO_NO_CAPACITY

termination_reason_message is None, so nothing indicates why. The task in step 4 provisions a second instance within ~20s against the same fleet, so the fleet itself is able to grow.

Expected behaviour

The fleet provisions a new instance and packs the job into a block, the same way jobs are packed onto blocks of an instance that already exists.

dstack version

master 1298b421a (source checkout)

Server logs

job filler-0-0: provisioning has started
job filler-0-0: Cannot combine fleet requirements
job filler-0-0: provisioning failed
Emitting event: Job status changed SUBMITTED -> TERMINATING. Termination reason: FAILED_TO_START_DUE_TO_NO_CAPACITY

Additional information

The run's requirements are intersected with the fleet's per-instance requirements when provisioning new capacity. A bare cpu: 1 parses as exactly 1..1, which does not intersect the fleet's 4..8, so the fleet yields no offers. Placement on an existing instance is unaffected because those offers are derived per block.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions