feat: Low Precision Allreduce for PCIe based GPU

[kanghui0204]

Low Precision Allreduce for PCIe based GPU

[feat] Support a new feature for low precision allreduce for PCIe based GPU

Description

This PR adds a customized allreduce to TensorRT-LLM. The new allreduce is used for communication on PCIe-based GPUs via low-precision quantization, which can accelerate the PCIe allreduce process.

Test Coverage

I added a unit test for this feature: tests/unittest/_torch/multi_gpu/test_lowprecision_allreduce.py. I hope this test can run on L40 or L20 nodes; otherwise, the low-precision allreduce will fall back to NCCL, making the test meaningless and causing it to fail.

[juney-nvidia]

@dongxuy04 @yuxianq Hi Dongxu, Yuxian, can you help review this MR?

Thanks
June

[kanghui0204]

/bot run --disable-fail-fast --add-multi-gpu-test

[hyukn]

Just some nits. LGTM.

[kanghui0204]

/bot run --disable-fail-fast --add-multi-gpu-test

[EmmaQiaoCh]

/bot run --disable-fail-fast --add-multi-gpu-test

[tensorrt-cicd]

PR_Github #4678 [ run ] triggered by Bot

[EmmaQiaoCh]

/bot run --stage-list "H100_PCIe-PyTorch-1"

[tensorrt-cicd]

PR_Github #4929 [ run ] triggered by Bot

[EmmaQiaoCh]

/bot run --disable-fail-fast --add-multi-gpu-test

[tensorrt-cicd]

PR_Github #4939 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #4929 [ run ] completed with state ABORTED

[tensorrt-cicd]

PR_Github #4939 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #3582 completed with status: 'FAILURE'

[hyukn]

/bot run --disable-fail-fast --only-multi-gpu-test

[tensorrt-cicd]

PR_Github #4980 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #4980 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #3617 (Partly Tested) completed with status: 'FAILURE'

[hyukn]

/bot run --disable-fail-fast --add-multi-gpu-test

[tensorrt-cicd]

PR_Github #5072 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #5072 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #3693 completed with status: 'FAILURE'

[EmmaQiaoCh]

/bot run --stage-list "B200_PCIe-PackageSanityCheck-DLFW"

[tensorrt-cicd]

PR_Github #5112 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #5112 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #3724 (Partly Tested) completed with status: 'SUCCESS'

[hyukn]

/bot reuse-pipeline

[tensorrt-cicd]

PR_Github #5150 [ reuse-pipeline ] triggered by Bot

[tensorrt-cicd]

PR_Github #5150 [ reuse-pipeline ] completed with state SUCCESS
Reusing PR_Github #5112 (Partly Tested) for commit 8614f2c

[QiJune]

Hi @kanghui0204 @hyukn , we found that multi-gpu test pipelines are not triggered as expected in this PR due to /bot reuse-pipeline. At the same time, the post merge multi-gpu test pipelines hangs. So, we revert this PR to fix broken CI first. Feel free to submit a new PR, and trigger multi-gpu test pipelines by /bot run --only-multi-gpu-test