can't run qwen3.6 27b in pipeline parallel

[competitiveNN]

Describe the bug
OVMS crashes as soon as I try to run 1 request
Hardware: 2x b580 (24GB)

To Reproduce
Steps to reproduce the behavior:
OpenVINO Model Server 2026.4.0.103456a0a
config.json

{
  "model_config_list": [
    {
      "config": {
        "name": "qwen3-6-27b-int4-ov_model",
        "base_path": "/models/ov/server/qwen3-6-27b-int4-ov",
        "target_device": "HETERO:GPU.0,GPU.1"
      }
    }
  ],
  "mediapipe_config_list": [
    {
      "name": "qwen3-6-27b-int4-ov",
      "graph_path": "/models/ov/server/qwen3-6-27b-int4-ov/graph.pbtxt"
    }
  ]
}

graph.pbxt

input_stream: "HTTP_REQUEST_PAYLOAD:input"
output_stream: "HTTP_RESPONSE_PAYLOAD:output"

node: {
  name: "LLMExecutor"
  calculator: "HttpLLMCalculator"
  input_stream: "LOOPBACK:loopback"
  input_stream: "HTTP_REQUEST_PAYLOAD:input"
  input_side_packet: "LLM_NODE_RESOURCES:llm"
  output_stream: "LOOPBACK:loopback"
  output_stream: "HTTP_RESPONSE_PAYLOAD:output"
  input_stream_info: {
    tag_index: 'LOOPBACK:0',
    back_edge: true
  }
  node_options: {
[type.googleapis.com / mediapipe.LLMCalculatorOptions]: {
    pipeline_type: VLM,
    models_path: "/models/ov/server/qwen3-6-27b-int4-ov/1",
    plugin_config: '{"DYNAMIC_QUANTIZATION_GROUP_SIZE": "32", "PERFORMANCE_HINT": "LATENCY", "KV_CACHE_PRECISION": "u8", "INFERENCE_PRECISION_HINT": "f16", "MODEL_DISTRIBUTION_POLICY": "PIPELINE_PARALLEL", "EXECUTION_MODE_HINT": "PERFORMANCE", "SCHEDULING_CORE_TYPE": "PCORE_ONLY", "ENABLE_CPU_PINNING": false, "DEVICE_PROPERTIES":{"GPU":{"MAX_PROMPT_LEN":2048}}}',
    enable_prefix_caching: false,
    cache_size: 1,
    max_num_batched_tokens: 2048,
    max_num_seqs: 1,
    device: "HETERO:GPU.0,GPU.1",
    tool_parser: "qwen3coder",
}
  }
  input_stream_handler {
    input_stream_handler: "SyncSetInputStreamHandler",
    options {
      [mediapipe.SyncSetInputStreamHandlerOptions.ext] {
        sync_set {
          tag_index: "LOOPBACK:0"
        }
      }
    }
  }
}

log.txt

[dtrawins]

@geunhwan Is model qwen3.6 27b supported in HETERO plugin now with A770 cards?

[mzegla]

I managed to reproduce that issue. The crash happens due to plugin config setting "MODEL_DISTRIBUTION_POLICY": "PIPELINE_PARALLEL".
even with single deployment:
ovms --model_repository_path /models --source_model OpenVINO/Qwen3.6-27B-int4-ov --model_name ovms-model --rest_port 9000 --target_device HETERO:GPU.1,GPU.2 --pipeline_type VLM --log_level DEBUG --plugin_config '{"MODEL_DISTRIBUTION_POLICY": "PIPELINE_PARALLEL"}'

Model works without plugin config parameter. Therefore this is not model server issue. The question about support for this configuration is valid, so let's wait for @geunhwan response.

[HumerousGorgon]

@geunhwan Is model qwen3.6 27b supported in HETERO plugin now with A770 cards?

I just tested on my A770's, it still doesn't work and tries to load the model into a single GPU.

[notsyncing]

Same here on 2x A770 16GB but on OpenVINO/Qwen3.8-27B-int4-ov model with ovms 2026.4 weekly docker image (de1d2eed43e2 . Model loads OK (model evenly distributed to the two cards with PIPELINE_PARALLEL config) but crashes on the first request without any error message (even with --log_level DEBUG).

But PIPELINE_PARALLEL works on both HETERO: GPU.0, CPU and HETERO: GPU.1, CPU, only crashes on HETERO: GPU.0, GPU.1.

dmesg log on crash:

[  934.026082] ovms[12478]: segfault at 707410173000 ip 0000707510408e09 sp 0000706d10486038 error 4 in libc.so.6[198e09,707510298000+188000] likely on CPU 1 (core 1, socket 0)
[  934.026097] Code: 07 00 01 74 a3 83 f9 c0 0f 87 59 fe ff ff 62 e1 fe 28 6f 4e 01 48 29 fe 48 83 c7 3f 49 8d 0c 10 48 83 e7 c0 48 01 fe 48 29 f9 <f3> a4 62 c1 fe 28 7f 00 62 c1 fe 28 7f 48 01 c3 0f 1f 80 00 00 00

ovms log: ovms-log.txt

[HumerousGorgon]

Same here on 2x A770 16GB but on OpenVINO/Qwen3.8-27B-int4-ov model with ovms 2026.4 weekly docker image (de1d2eed43e2 . Model loads OK (model evenly distributed to the two cards with PIPELINE_PARALLEL config) but crashes on the first request without any error message (even with --log_level DEBUG).

But PIPELINE_PARALLEL works on both HETERO: GPU.0, CPU and HETERO: GPU.1, CPU, only crashes on HETERO: GPU.0, GPU.1.

dmesg log on crash:

[  934.026082] ovms[12478]: segfault at 707410173000 ip 0000707510408e09 sp 0000706d10486038 error 4 in libc.so.6[198e09,707510298000+188000] likely on CPU 1 (core 1, socket 0)
[  934.026097] Code: 07 00 01 74 a3 83 f9 c0 0f 87 59 fe ff ff 62 e1 fe 28 6f 4e 01 48 29 fe 48 83 c7 3f 49 8d 0c 10 48 83 e7 c0 48 01 fe 48 29 f9 <f3> a4 62 c1 fe 28 7f 00 62 c1 fe 28 7f 48 01 c3 0f 1f 80 00 00 00

ovms log: ovms-log.txt

Guarentee you it’s the same problem that was happening months ago; openvino doesn’t split the model nor the kv cache across the gpus even though it knows there are multiple gpus in the hetero configuration.

[notsyncing]

Guarentee you it’s the same problem that was happening months ago; openvino doesn’t split the model nor the kv cache across the gpus even though it knows there are multiple gpus in the hetero configuration.

No, in my situation, both GPU got ~9GB VRAM usage with PIPELINE_PARALLEL from xpu-smi, looks like the split works. If you do not specify PIPELINE_PARALLEL, the first GPU will got the full model weight (16GB+) and easily OOM.

[notsyncing]

I made qwen3.8 27b work on 2x A770 HETERO PIPELINE_PARALLEL with these patches for OVMS and GenAI, all patches are generated by AI (OpenCode Big Pickle), based on current main branch:

OVMS patch:

0000-ovms.patch Fixes the crash on the first request issue. Looks like mainly a threading issue. Ignore the dockerfile modification if it doesn't match your environment.

GenAI patch:

0001-CB-Preallocate-KV-and-linear-attention-cache-to-conf.patch Workarounds CL_EXEC_STATUS_ERROR when prefilling.

0002-CB-LinearAttention-Fix-garbled-generation-on-GPU-sto.patch Fixes garbled outputs (like this: 2343#!/23283##%) starting from the second token after some conversations.
I also have some questions about this patch: AI states that the per-block CPU -> RemoteTensor ROI copies will silently corrupt the zeroed memory block, which I feel strange. Is there some race conditions when copying? Or this only happens on A770? I tried update the compute-runtime from 26.18 to 26.31 but it still generates grabled outputs.

0003-CB-Sum-GPU-memory-across-devices-for-multi-GPU-KV-ca.patch PR #4254 or it will OOM when VRAM of the first GPU is full.

0004-CB-LinearAttention-Preallocate-bounded-LA-state-so-t.patch Workarounds CL_EXEC_STATUS_ERROR (-14) via clWaitForEvents when prefilling, completing the first patch.
Still tensor copying issues. I guess the real problem sits in some other place.

Qwen3.8-27b-ov-int4 now works on my 2x A770 16GB setup, reaching ~192k context with 7GB cache. OVMS arguments are:

--model_name Qwen3.8-27B
--model_path /models/OpenVINO/Qwen3.8-27B-int4-ov
--task text_generation
--pipeline_type VLM_CB
--target_device HETERO:GPU.0,GPU.1
--model_distribution_policy PIPELINE_PARALLEL
--plugin_config '{"GPU_ENABLE_LARGE_ALLOCATIONS": true}'
--cache_size 7
--rest_port 9000

Dynamic cache will result in CL_OUT_OF_RESOURCES when approaching ~160k context.

The performance is good, tested by llama-benchy:

model	test	t/s	peak t/s	ttfr (ms)	est_ppt (ms)	e2e_ttft (ms)
Qwen/Qwen3.8-27B	pp512	614.96 ± 86.37		851.18 ± 109.20	849.00 ± 109.20	1003.91 ± 29.98
Qwen/Qwen3.8-27B	tg128	14.91 ± 0.02	15.00 ± 0.00
Qwen/Qwen3.8-27B	pp512 @ d4096	777.57 ± 0.35		5928.36 ± 2.67	5926.18 ± 2.67	6196.09 ± 3.43
Qwen/Qwen3.8-27B	tg128 @ d4096	14.64 ± 0.03	15.00 ± 0.00
Qwen/Qwen3.8-27B	pp512 @ d8192	725.09 ± 7.08		12008.20 ± 117.13	12006.03 ± 117.13	12169.54 ± 35.76
Qwen/Qwen3.8-27B	tg128 @ d8192	13.98 ± 0.00	14.00 ± 0.00
Qwen/Qwen3.8-27B	pp512 @ d16384	640.40 ± 10.08		26394.91 ± 420.48	26392.73 ± 420.48	26551.31 ± 443.40
Qwen/Qwen3.8-27B	tg128 @ d16384	13.16 ± 0.01	14.00 ± 0.00
Qwen/Qwen3.8-27B	pp512 @ d65536	362.65 ± 0.16		182131.29 ± 82.29	182129.11 ± 82.29	182331.79 ± 80.80
Qwen/Qwen3.8-27B	tg128 @ d65536	9.78 ± 0.02	10.00 ± 0.00
Qwen/Qwen3.8-27B	pp512 @ d128000	236.06 ± 0.24		544414.67 ± 543.05	544412.49 ± 543.05	545030.46 ± 473.33
Qwen/Qwen3.8-27B	tg128 @ d128000	7.29 ± 0.02	8.00 ± 0.00

llama-benchy (0.4.0)
date: 2026-09-01 22:00:39 | latency mode: api

The GPU utilizations are under 15% on both cards, so I guess the A770 may have some potentials.

These patches are fully vibe-coded and not ready for a formal PR. I wish they could be a reference for a real fix. OpenVINO is currently the ONLY performant solution on Intel ARC consumer GPUs, specially on old cards like A770. So much thanks to the OpenVINO team!