Qwen hybrid attention with built-in multi-token prediction exposes three gaps on NVIDIA A6000 (SM86):
- The ATen adaptor has no
DataType::F8 / DataType::BOOL mapping, so E4M3 weight conversion and device acceptance-mask casts cannot use the existing bridge.
ncclCommDestroy may change the current CUDA device, leaving the caller's existing stream/context inconsistent during sequential TP2 engine shutdown and reconstruction.- Paged Prefill for head size 256 defaults to the reference kernel although the existing warp implementation can support eight dimensions per lane.
The proposed small patch reuses the existing casts and warp kernel, restores the caller's device after communicator teardown, and extends core correctness tests. NVIDIA-only dispatch changes must leave unvalidated vendor defaults intact. Target branch: InfiniLM-v0.2.9c.
Implemented in #1566; associated model integration: InfiniTensor/InfiniLM#584.
Validation completed on an isolated build from the target branch: 88 paged Prefill cases, 2 finite E4M3/BOOL cast tests, and communicator device preservation from caller devices 0/1. InfiniLM TP1/TP2 integration and real 27B FP8 lifecycle checks also passed using the matching combined runtime. Graph lifetime changes remain in the separate PR #1560. Detailed conditions and logs are attached to the PRs.
Qwen hybrid attention with built-in multi-token prediction exposes three gaps on NVIDIA A6000 (SM86):
DataType::F8/DataType::BOOLmapping, so E4M3 weight conversion and device acceptance-mask casts cannot use the existing bridge.ncclCommDestroymay change the current CUDA device, leaving the caller's existing stream/context inconsistent during sequential TP2 engine shutdown and reconstruction.The proposed small patch reuses the existing casts and warp kernel, restores the caller's device after communicator teardown, and extends core correctness tests. NVIDIA-only dispatch changes must leave unvalidated vendor defaults intact. Target branch:
InfiniLM-v0.2.9c.Implemented in #1566; associated model integration: InfiniTensor/InfiniLM#584.
Validation completed on an isolated build from the target branch: 88 paged Prefill cases, 2 finite E4M3/BOOL cast tests, and communicator device preservation from caller devices 0/1. InfiniLM TP1/TP2 integration and real 27B FP8 lifecycle checks also passed using the matching combined runtime. Graph lifetime changes remain in the separate PR #1560. Detailed conditions and logs are attached to the PRs.