Skip to content

[release/2.7][ROCm][TunableOp] TunableOp TF32 support #2049

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 4 commits into
base: release/2.7
Choose a base branch
from
Open

[release/2.7][ROCm][TunableOp] TunableOp TF32 support #2049

wants to merge 4 commits into from
+227 −4

Conversation

naromero77amd
Copy link

TunableOp TF32 support enabled via two-cherry picks from upstream.

pytorch#145946
pytorch#149088

jataylo and others added 4 commits April 18, 2025 21:37
@jataylo @jithunnair-amd
Update triton.txt to point to internal ROCm branch (#2028)
9ec21e4 
https://github.com/ROCm/triton/commits/release/internal/3.3.x/
@jithunnair-amd
Update triton commit with cherry-picks
2b1f8dd
@jeffdaily @naromero77amd
[ROCm][TunableOp] hipblaslt tf32 support (pytorch#145946)
14d54c7 
TF32 is supported by hipblaslt. Support added by pytorch#143549.  This PR expands integration to the TunableOp feature.

Pull Request resolved: pytorch#145946
Approved by: https://github.com/pruthvistony, https://github.com/echen4096, https://github.com/yoyoyocmu

Co-authored-by: Nichols A. Romero <[email protected]>
(cherry picked from commit 0c8ec26)
@naromero77amd
[ROCm][TunableOp] More TF32 support. (pytorch#149088)
0514332 
This PR includes additional enhancements to TF32 support in TunableOp.
- OpSignature now differentiates between float32 and tf32 data types.
- Offline tuning now supports TF32.
- Unit tests for online and offline tuning of TF32.

Pull Request resolved: pytorch#149088
Approved by: https://github.com/jeffdaily

Co-authored-by: Jeff Daily <[email protected]>
(cherry picked from commit 11d4438)
@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Apr 23, 2025 •
edited
Loading

Jenkins build for 0514332de8a5c32775566c9ef5c7b2063280deff commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@naromero77amd
Copy link
Author

This was a feature that just missed the PyTorch 2.7 cutoff date. Jeff said that it would be worth backporting since some customers are asking for it.

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Apr 24, 2025 •
edited
Loading

Jenkins build for 0514332de8a5c32775566c9ef5c7b2063280deff commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

Detected error during Pytorch building:

/var/lib/jenkins/pytorch/aten/src/ATen/native/hip/HIPLoops.cuh:181:17: warning: loop not unrolled: the optimizer was unable to perform the requested transformation; the transformation might be disabled or specified as part of an unsupported transformation ordering [-Wpass-failed=transform-warning]
40 warnings generated when compiling for gfx942.
[6962/7809] Building HIPCC object caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/torch_hip_generated_GcdLcmKernel.hip.o
clang++: warning: argument unused during compilation: '--offload-compress' [-Wunused-command-line-argument]
[6963/7809] Building HIPCC object caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/torch_hip_generated_Reduce.hip.o
FAILED: caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/torch_hip_generated_Reduce.hip.o /var/lib/jenkins/pytorch/build/caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/torch_hip_generated_Reduce.hip.o 
cd /var/lib/jenkins/pytorch/build/caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip && /opt/conda/envs/py_3.10/bin/cmake -E make_directory /var/lib/jenkins/pytorch/build/caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/. && /opt/conda/envs/py_3.10/bin/cmake -D verbose:BOOL=OFF -D build_configuration:STRING=RELEASE -D generated_file:STRING=/var/lib/jenkins/pytorch/build/caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/./torch_hip_generated_Reduce.hip.o -P /var/lib/jenkins/pytorch/build/caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/torch_hip_generated_Reduce.hip.o.cmake
clang++: warning: argument unused during compilation: '--offload-compress' [-Wunused-command-line-argument]
sccache: encountered fatal error
sccache: error: Failed to parse included file path
sccache: caused by: Failed to parse included file path

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Apr 24, 2025 •
edited
Loading

Jenkins build for 0514332de8a5c32775566c9ef5c7b2063280deff commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@naromero77amd
Copy link
Author

FWIW, ran all tunableop tests locally on MI300 and they all pass.

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Apr 25, 2025 •
edited
Loading

Jenkins build for 0514332de8a5c32775566c9ef5c7b2063280deff commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@jithunnair-amd jithunnair-amd requested a review from jataylo as a code owner April 25, 2025 07:57
@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Apr 25, 2025 •
edited
Loading

Jenkins build for 0514332de8a5c32775566c9ef5c7b2063280deff commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Apr 25, 2025 •
edited
Loading

Jenkins build for 0514332de8a5c32775566c9ef5c7b2063280deff commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pruthvistony pruthvistony pruthvistony approved these changes

@jeffdaily jeffdaily Awaiting requested review from jeffdaily jeffdaily is a code owner

@jithunnair-amd jithunnair-amd Awaiting requested review from jithunnair-amd jithunnair-amd is a code owner

@jataylo jataylo Awaiting requested review from jataylo jataylo is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@naromero77amd @pruthvistony @jataylo @jithunnair-amd @jeffdaily