Skip to content

[ET-VK] Use shared pointer for vTensorStorage #11487

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Jun 10, 2025
Merged

[ET-VK] Use shared pointer for vTensorStorage #11487

merged 2 commits into from
Jun 10, 2025

Conversation

pytorchbot
Copy link
Collaborator

This PR was created by the merge bot to help merge the original PR into the main branch.
ghstack PR number: #11400 by @SS-JIA
^ Please use this as the source of truth for the PR details, comments, and reviews
ghstack PR base: https://github.com/pytorch/executorch/tree/gh/SS-JIA/236/base
ghstack PR head: https://github.com/pytorch/executorch/tree/gh/SS-JIA/236/head
Merge bot PR base: https://github.com/pytorch/executorch/tree/gh/SS-JIA/238/orig
Merge bot PR head: https://github.com/pytorch/executorch/tree/gh/SS-JIA/236/orig
@diff-train-skip-merge

SS-JIA added 2 commits June 9, 2025 07:38
@SS-JIA
[ET-VK][ez][testing] Remove conv2d test in compute_api_test + reduce …
30733fd 
…stdout logging

Pull Request resolved: #11452

## Changes

Some changes to `vulkan_compute_api_test`:

* Remove `conv2d_prepack_test`. D74523774 / #11306 recently changed the conv2d prepack shader to use push constants, and since this test uses the "old" vulkan compute API (which doesn't support push constants) the test broke. Operator tests at this point are handled by `compute_graph_op_tests` anyway so just remove this test.
* Make it so that verbose logging in `test_to_copy` only occurs in `VULKAN_DEBUG` is defined
* Disable `test_etvk_copy_channel_offset_node*` tests, as these tests fail on some platforms (i.e. devGPU, windows, mac) likely due to an error in the implementation of the op.
ghstack-source-id: 289110007
@exported-using-ghexport

Differential Revision: [D76156109](https://our.internmc.facebook.com/intern/diff/D76156109/)
@SS-JIA
[ET-VK] Use shared pointer for vTensorStorage
5100451 
Pull Request resolved: #11400

## Changes

* In `vTensor`, store the `storage_` member using a shared pointer instead of a direct `vTensorStorage` object
* Remove the ability to construct a tensor view with a buffer offset

## Motivation

> * In `vTensor`, store the `storage_` member using a shared pointer instead of a direct `vTensorStorage` object

Previously, to support the ability to create tensor views I implemented copy constructors for `VulkanImage`, `VulkanBuffer`, and `vTensorStorage`. The idea was that the copied object instance would have the same handles as the original, but would not own the handle (and therefore not be responsible for destroying it).

However, the problem with this approach is that in order to construct pipeline barriers and  image layout transitions properly, we must know the details of how a resource was last accessed. Since tensor views make copies, accessing the original tensor will not update the access information of any copies it may have. Therefore, if the copy is then accessed, the last access information it stores would then be out of date.

I originally tried to solve this problem with crude assumptions in the `transition()` function, but unfortunately the solution was not robust enough. I discovered validation errors such as

```
[ RUN      ] VulkanComputeGraphOpsTest.test_transpose_with_mm
VUID-VkImageMemoryBarrier-oldLayout-01197(ERROR / SPEC): msgNum: 307231540 - Validation Error: [ VUID-VkImageMemoryBarrier-oldLayout-01197 ] Object 0: handle = 0x21eba07e160, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x35643f0000000111, type = VK_OBJECT_TYPE_IMAGE; | MessageID = 0x124ffb34 | vkCmdPipelineBarrier(): pImageMemoryBarriers[2].image (VkImage 0x35643f0000000111[]) cannot transition the layout of aspect=1, level=0, layer=0 from VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL when the previous known layout is VK_IMAGE_LAYOUT_GENERAL.
The Vulkan spec states: If srcQueueFamilyIndex and dstQueueFamilyIndex define a queue family ownership transfer or oldLayout and newLayout define an image layout transition, oldLayout must be VK_IMAGE_LAYOUT_UNDEFINED or the current layout of the image subresources affected by the barrier (https://vulkan.lunarg.com/doc/view/1.3.296.0/windows/1.3-extensions/vkspec.html#VUID-VkImageMemoryBarrier-oldLayout-01197)
    Objects: 2
        [0] 0x21eba07e160, type: 6, name: NULL
        [1] 0x35643f0000000111, type: 10, name: NULL
UNASSIGNED-CoreValidation-DrawState-InvalidImageLayout(ERROR / SPEC): msgNum: 1303270965 - Validation Error: [ UNASSIGNED-CoreValidation-DrawState-InvalidImageLayout ] Object 0: handle = 0x21eba07e160, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x35643f0000000111, type = VK_OBJECT_TYPE_IMAGE; | MessageID = 0x4dae5635 | vkQueueSubmit(): pSubmits[0].pCommandBuffers[0] command buffer VkCommandBuffer 0x21eba07e160[] expects VkImage 0x35643f0000000111[] (subresource: aspectMask VK_IMAGE_ASPECT_COLOR_BIT array layer 0, mip level 0) to be in layout VK_IMAGE_LAYOUT_GENERAL--instead, current layout is VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL.
    Objects: 2
        [0] 0x21eba07e160, type: 6, name: NULL
        [1] 0x35643f0000000111, type: 10, name: NULL
VUID-VkImageMemoryBarrier-oldLayout-01197(ERROR / SPEC): msgNum: 307231540 - Validation Error: [ VUID-VkImageMemoryBarrier-oldLayout-01197 ] Object 0: handle = 0x21eba07e160, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x35643f0000000111, type = VK_OBJECT_TYPE_IMAGE; | MessageID = 0x124ffb34 | vkCmdPipelineBarrier(): pImageMemoryBarriers[2].image (VkImage 0x35643f0000000111[]) cannot transition the layout of aspect=1, level=0, layer=0 from VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL when the previous known layout is VK_IMAGE_LAYOUT_GENERAL.
The Vulkan spec states: If srcQueueFamilyIndex and dstQueueFamilyIndex define a queue family ownership transfer or oldLayout and newLayout define an image layout transition, oldLayout must be VK_IMAGE_LAYOUT_UNDEFINED or the current layout of the image subresources affected by the barrier (https://vulkan.lunarg.com/doc/view/1.3.296.0/windows/1.3-extensions/vkspec.html#VUID-VkImageMemoryBarrier-oldLayout-01197)
    Objects: 2
        [0] 0x21eba07e160, type: 6, name: NULL
        [1] 0x35643f0000000111, type: 10, name: NULL
UNASSIGNED-CoreValidation-DrawState-InvalidImageLayout(ERROR / SPEC): msgNum: 1303270965 - Validation Error: [ UNASSIGNED-CoreValidation-DrawState-InvalidImageLayout ] Object 0: handle = 0x21eba07e160, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x35643f0000000111, type = VK_OBJECT_TYPE_IMAGE; | MessageID = 0x4dae5635 | vkQueueSubmit(): pSubmits[0].pCommandBuffers[0] command buffer VkCommandBuffer 0x21eba07e160[] expects VkImage 0x35643f0000000111[] (subresource: aspectMask VK_IMAGE_ASPECT_COLOR_BIT array layer 0, mip level 0) to be in layout VK_IMAGE_LAYOUT_GENERAL--instead, current layout is VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL.
    Objects: 2
        [0] 0x21eba07e160, type: 6, name: NULL
        [1] 0x35643f0000000111, type: 10, name: NULL
VUID-VkImageMemoryBarrier-oldLayout-01197(ERROR / SPEC): msgNum: 307231540 - Validation Error: [ VUID-VkImageMemoryBarrier-oldLayout-01197 ] Object 0: handle = 0x21eba07e160, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x35643f0000000111, type = VK_OBJECT_TYPE_IMAGE; | MessageID = 0x124ffb34 | vkCmdPipelineBarrier(): pImageMemoryBarriers[2].image (VkImage 0x35643f0000000111[]) cannot transition the layout of aspect=1, level=0, layer=0 from VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL when the previous known layout is VK_IMAGE_LAYOUT_GENERAL.
The Vulkan spec states: If srcQueueFamilyIndex and dstQueueFamilyIndex define a queue family ownership transfer or oldLayout and newLayout define an image layout transition, oldLayout must be VK_IMAGE_LAYOUT_UNDEFINED or the current layout of the image subresources affected by the barrier (https://vulkan.lunarg.com/doc/view/1.3.296.0/windows/1.3-extensions/vkspec.html#VUID-VkImageMemoryBarrier-oldLayout-01197)
    Objects: 2
        [0] 0x21eba07e160, type: 6, name: NULL
        [1] 0x35643f0000000111, type: 10, name: NULL
UNASSIGNED-CoreValidation-DrawState-InvalidImageLayout(ERROR / SPEC): msgNum: 1303270965 - Validation Error: [ UNASSIGNED-CoreValidation-DrawState-InvalidImageLayout ] Object 0: handle = 0x21eba07e160, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x35643f0000000111, type = VK_OBJECT_TYPE_IMAGE; | MessageID = 0x4dae5635 | vkQueueSubmit(): pSubmits[0].pCommandBuffers[0] command buffer VkCommandBuffer 0x21eba07e160[] expects VkImage 0x35643f0000000111[] (subresource: aspectMask VK_IMAGE_ASPECT_COLOR_BIT array layer 0, mip level 0) to be in layout VK_IMAGE_LAYOUT_GENERAL--instead, current layout is VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL.
    Objects: 2
        [0] 0x21eba07e160, type: 6, name: NULL
        [1] 0x35643f0000000111, type: 10, name: NULL
VUID-VkImageMemoryBarrier-oldLayout-01197(ERROR / SPEC): msgNum: 307231540 - Validation Error: [ VUID-VkImageMemoryBarrier-oldLayout-01197 ] Object 0: handle = 0x21eba07e160, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x35643f0000000111, type = VK_OBJECT_TYPE_IMAGE; | MessageID = 0x124ffb34 | vkCmdPipelineBarrier(): pImageMemoryBarriers[2].image (VkImage 0x35643f0000000111[]) cannot transition the layout of aspect=1, level=0, layer=0 from VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL when the previous known layout is VK_IMAGE_LAYOUT_GENERAL.
The Vulkan spec states: If srcQueueFamilyIndex and dstQueueFamilyIndex define a queue family ownership transfer or oldLayout and newLayout define an image layout transition, oldLayout must be VK_IMAGE_LAYOUT_UNDEFINED or the current layout of the image subresources affected by the barrier (https://vulkan.lunarg.com/doc/view/1.3.296.0/windows/1.3-extensions/vkspec.html#VUID-VkImageMemoryBarrier-oldLayout-01197)
    Objects: 2
        [0] 0x21eba07e160, type: 6, name: NULL
        [1] 0x35643f0000000111, type: 10, name: NULL
UNASSIGNED-CoreValidation-DrawState-InvalidImageLayout(ERROR / SPEC): msgNum: 1303270965 - Validation Error: [ UNASSIGNED-CoreValidation-DrawState-InvalidImageLayout ] Object 0: handle = 0x21eba07e160, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x35643f0000000111, type = VK_OBJECT_TYPE_IMAGE; | MessageID = 0x4dae5635 | vkQueueSubmit(): pSubmits[0].pCommandBuffers[0] command buffer VkCommandBuffer 0x21eba07e160[] expects VkImage 0x35643f0000000111[] (subresource: aspectMask VK_IMAGE_ASPECT_COLOR_BIT array layer 0, mip level 0) to be in layout VK_IMAGE_LAYOUT_GENERAL--instead, current layout is VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL.
    Objects: 2
        [0] 0x21eba07e160, type: 6, name: NULL
        [1] 0x35643f0000000111, type: 10, name: NULL
VUID-VkImageMemoryBarrier-oldLayout-01197(ERROR / SPEC): msgNum: 307231540 - Validation Error: [ VUID-VkImageMemoryBarrier-oldLayout-01197 ] Object 0: handle = 0x21eba07e160, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x35643f0000000111, type = VK_OBJECT_TYPE_IMAGE; | MessageID = 0x124ffb34 | vkCmdPipelineBarrier(): pImageMemoryBarriers[2].image (VkImage 0x35643f0000000111[]) cannot transition the layout of aspect=1, level=0, layer=0 from VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL when the previous known layout is VK_IMAGE_LAYOUT_GENERAL.
The Vulkan spec states: If srcQueueFamilyIndex and dstQueueFamilyIndex define a queue family ownership transfer or oldLayout and newLayout define an image layout transition, oldLayout must be VK_IMAGE_LAYOUT_UNDEFINED or the current layout of the image subresources affected by the barrier (https://vulkan.lunarg.com/doc/view/1.3.296.0/windows/1.3-extensions/vkspec.html#VUID-VkImageMemoryBarrier-oldLayout-01197)
    Objects: 2
        [0] 0x21eba07e160, type: 6, name: NULL
        [1] 0x35643f0000000111, type: 10, name: NULL
UNASSIGNED-CoreValidation-DrawState-InvalidImageLayout(ERROR / SPEC): msgNum: 1303270965 - Validation Error: [ UNASSIGNED-CoreValidation-DrawState-InvalidImageLayout ] Object 0: handle = 0x21eba07e160, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0x35643f0000000111, type = VK_OBJECT_TYPE_IMAGE; | MessageID = 0x4dae5635 | vkQueueSubmit(): pSubmits[0].pCommandBuffers[0] command buffer VkCommandBuffer 0x21eba07e160[] expects VkImage 0x35643f0000000111[] (subresource: aspectMask VK_IMAGE_ASPECT_COLOR_BIT array layer 0, mip level 0) to be in layout VK_IMAGE_LAYOUT_GENERAL--instead, current layout is VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL.
    Objects: 2
        [0] 0x21eba07e160, type: 6, name: NULL
        [1] 0x35643f0000000111, type: 10, name: NULL
[       OK ] VulkanComputeGraphOpsTest.test_transpose_with_mm (26 ms)
```

when using tensor views. The simplest solution is to use a shared pointer to store the tensor storage object, that way tensor views can point to the same underlying object instance, and last access metadata will be consistent among all tensor views.

> * Remove the ability to construct a tensor view with a buffer offset

Removed this because it turns out that buffer properties cannot have arbitrary offsets; there are restrictions for what offsets can be used. The validation errors say it all:

```
[ RUN      ] VulkanComputeGraphTest.test_simple_graph_with_view
VUID-VkWriteDescriptorSet-descriptorType-00328(ERROR / SPEC): msgNum: -368569266 - Validation Error: [ VUID-VkWriteDescriptorSet-descriptorType-00328 ] | MessageID = 0xea08144e | vkUpdateDescriptorSets(): pDescriptorWrites[1].pBufferInfo[0].offset (84) must be a multiple of device limit minStorageBufferOffsetAlignment 16 when descriptor type is VK_DESCRIPTOR_TYPE_STORAGE_BUFFER.
The Vulkan spec states: If descriptorType is VK_DESCRIPTOR_TYPE_STORAGE_BUFFER or VK_DESCRIPTOR_TYPE_STORAGE_BUFFER_DYNAMIC, the offset member of each element of pBufferInfo must be a multiple of VkPhysicalDeviceLimits::minStorageBufferOffsetAlignment (https://vulkan.lunarg.com/doc/view/1.3.296.0/windows/1.3-extensions/vkspec.html#VUID-VkWriteDescriptorSet-descriptorType-00328)
VUID-VkDescriptorBufferInfo-range-00342(ERROR / SPEC): msgNum: -371195848 - Validation Error: [ VUID-VkDescriptorBufferInfo-range-00342 ] Object 0: handle = 0xee647e0000000009, type = VK_OBJECT_TYPE_BUFFER; | MessageID = 0xe9e00038 | vkUpdateDescriptorSets(): pDescriptorWrites[1].pBufferInfo[0].range (224) is larger than buffer size (224) + offset (84).
The Vulkan spec states: If range is not equal to VK_WHOLE_SIZE, range must be less than or equal to the size of buffer minus offset (https://vulkan.lunarg.com/doc/view/1.3.296.0/windows/1.3-extensions/vkspec.html#VUID-VkDescriptorBufferInfo-range-00342)
    Objects: 1
        [0] 0xee647e0000000009, type: 9, name: NULL
VUID-VkBufferMemoryBarrier-size-01189(ERROR / SPEC): msgNum: -1238074894 - Validation Error: [ VUID-VkBufferMemoryBarrier-size-01189 ] Object 0: handle = 0x25d7d900870, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID = 0xb63479f2 | vkCmdPipelineBarrier(): pBufferMemoryBarriers[0].size VkBuffer 0xee647e0000000009[] has offset 0x54 and size 0xe0 whose sum is greater than total size 0xe0.
The Vulkan spec states: If size is not equal to VK_WHOLE_SIZE, size must be less than or equal to than the size of buffer minus offset (https://vulkan.lunarg.com/doc/view/1.3.296.0/windows/1.3-extensions/vkspec.html#VUID-VkBufferMemoryBarrier-size-01189)
    Objects: 1
        [0] 0x25d7d900870, type: 6, name: NULL
[       OK ] VulkanComputeGraphTest.test_simple_graph_with_view (8 ms)
```

## Impact

* Heap allocation upon tensor construction
* Increased pointer chasing when accessing `vTensor` members

I will validate that the impact of this change does not regress load/inference latency.
ghstack-source-id: 289110003

Differential Revision: [D76047204](https://our.internmc.facebook.com/intern/diff/D76047204/)
@pytorchbot pytorchbot requested a review from SS-JIA as a code owner June 9, 2025 16:35
@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Jun 9, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/11487

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 5100451 with merge base ef1d2ff (image):

NEW FAILURE - The following job has failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jun 9, 2025
Base automatically changed from gh/SS-JIA/238/orig to main June 10, 2025 00:48
@SS-JIA SS-JIA merged commit f3fde3c into main Jun 10, 2025
94 of 95 checks passed
@SS-JIA SS-JIA deleted the gh/SS-JIA/236/orig branch June 10, 2025 00:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SS-JIA SS-JIA SS-JIA approved these changes

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@pytorchbot @SS-JIA @facebook-github-bot