Full dynamic shapes support design

[CharlieL7]

Discussion of the current design for dynamic batch support and how it might be extended for full dynamic shape support.

[CharlieL7]

@qianqing13579 I'm writing up a document that I will put in this repository's Wiki page about the current design of dynamic shapes in MIGX. We can then discuss the plan fowards.

[CharlieL7]

Here is the Wiki page https://github.com/ROCmSoftwarePlatform/AMDMIGraphX/wiki/Dynamic-Shapes-Design.

[CharlieL7]

The dynamic batch design that splits each size into submodules will only work for one dynamic dimension. This is intentional. For operators with a data-dependent shape function we will either have to use dynamic GPU kernels or change the instructions such that it works with zero padding. There is also the option to use the ref target for these operators and subsequent ones to get something "working".

[qianqing13579]

@CharlieL7 ，How to run resnet50 ONNX model with dynamic batch? can you give me a sample?

[CharlieL7]

Yes, here's a example using the C++ API that should be easy to changed to run resnet50: https://github.com/ROCmSoftwarePlatform/AMDMIGraphX/tree/develop/examples/migraphx/cpp_dynamic_batch. Let me know if you have any questions.

[qianqing13579]

@CharlieL7 ,resnet18 can run successfully. but yolov4 model can not run successfully. can the current design for dynamic batch support the data-dependent op?
the yolov4 model link : https://github.com/onnx/models/blob/main/vision/object_detection_segmentation/yolov4/model/yolov4.onnx

[CharlieL7]

Data-dependent operators will have to be handled separately from dynamic batch. It's possible to handle with the same idea, but would be limiting. We're working on updates to get yolov4 and transformer based models working with dynamic batch. For yolov4 we're updating how we're handling Resize from ONNX.

[yiakwy-xpu-ml-framework-team]

The current dynamic batch solution means multiple copies in GPU : that is impossible for LLM models in LLM era.

Let's forget about this solution (this is designed for ASIC which compile model into chip in advance, in which case batch size must be fixed number).

GPU is indead naturally support dyanmic batches : activations can be allocated when produced and removed when data dependecy goes to zero!

Unlike General purpose ASIC such as Graphcore IPU chip, you have to reserve memory in compile stage (stack memory), while GPU ASIC can request memory just in need (HBM memory pool dynamically).

Hence the solution should turn to memory dynamically allocation for actiations .

Note you cannot perfectly infer shape from an arbitrary model with Reshape and Resize ops (this has been proved) when you don't know the batch size.

@CharlieL7

[CharlieL7]

Yes, this is possible with GPU kernels that support dynamic shapes. Many of the current graph optimizations and fusions will not be doable with dynamic kernels however. We have not yet planned to implement that feature. We're open to code contributions for this though!