Open
Description
🐛 Describe the bug
tinyllama+mps has been failing consistently: https://github.com/pytorch/executorch/actions/workflows/apple-perf.yml?query=event%3Aschedule
We will disable this model from continuous benchmarking and track the bug/fix here.
The stacktrace:
Test Case '-[GenericTests test_forward_tinyllama_mps_fp32_pte_iOS_17_2_1_iPhone15_4]' passed (19.757 seconds).
Test Case '-[GenericTests test_load_tinyllama_mps_fp32_pte_iOS_17_2_1_iPhone15_4]' started.
2025-01-22 22:03:20.769704-0800 Benchmark[554:39734] Invalid group id in source layers: test_layer
2025-01-22 22:03:20.769798-0800 Benchmark[554:39734] Validation failure: Tensor dimensions N1D1C768H1W32000 are not within supported range, N[1-65536]D[1-16384]C[1-65536]H[1-16384]W[1-16384].
loc("mps_transpose_84"("(mpsFileLoc): /Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":206:0)): error: 'anec.transpose' op Invalid configuration for the following reasons: Tensor dimensions N1D1C768H1W32000 are not within supported range, N[1-65536]D[1-16384]C[1-65536]H[1-16384]W[1-16384].
2025-01-22 22:03:20.984004-0800 Benchmark[554:39734] Invalid group id in source layers: test_layer
2025-01-22 22:03:20.984099-0800 Benchmark[554:39734] Validation failure: Tensor dimensions N1D1C768H1W32000 are not within supported range, N[1-65536]D[1-16384]C[1-65536]H[1-16384]W[1-16384].
loc("mps_matmul_108"("(mpsFileLoc): /Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":39:0)): error: 'anec.matmul' op Invalid configuration for the following reasons: Tensor dimensions N1D1C768H1W32000 are not within supported range, N[1-65536]D[1-16384]C[1-65536]H[1-16384]W[1-16384].
loc("mps_broadcast_to"("(mpsFileLoc): /Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":1327:0)): error: 'anec.broadcast' op failed: input cannot be broadcasted to the target shape
loc("mps_broadcast_to_2"("(mpsFileLoc): /Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":1327:0)): error: 'anec.broadcast' op failed: input cannot be broadcasted to the target shape
loc("mps_broadcast_to_4"("(mpsFileLoc): /Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":1327:0)): error: 'anec.broadcast' op failed: input cannot be broadcasted to the target shape
loc("mps_broadcast_to_6"("(mpsFileLoc): /Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":1327:0)): error: 'anec.broadcast' op failed: input cannot be broadcasted to the target shape
loc("mps_broadcast_to_8"("(mpsFileLoc): /Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":1327:0)): error: 'anec.broadcast' op failed: input cannot be broadcasted to the target shape
loc("mps_broadcast_to_10"("(mpsFileLoc): /Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":1327:0)): error: 'anec.broadcast' op failed: input cannot be broadcasted to the target shape
loc("mps_broadcast_to_12"("(mpsFileLoc): /Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":1327:0)): error: 'anec.broadcast' op failed: input cannot be broadcasted to the target shape
loc("mps_broadcast_to_14"("(mpsFileLoc): /Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":1327:0)): error: 'anec.broadcast' op failed: input cannot be broadcasted to the target shape
loc("mps_broadcast_to_16"("(mpsFileLoc): /Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":1327:0)): error: 'anec.broadcast' op failed: input cannot be broadcasted to the target shape
loc("mps_broadcast_to_18"("(mpsFileLoc): /Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":1327:0)): error: 'anec.broadcast' op failed: input cannot be broadcasted to the target shape
loc("mps_broadcast_to_20"("(mpsFileLoc): /Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":1327:0)): error: 'anec.broadcast' op failed: input cannot be broadcasted to the target shape
loc("mps_broadcast_to_22"("(mpsFileLoc): /Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":1327:0)): error: 'anec.broadcast' op failed: input cannot be broadcasted to the target shape
2025-01-22 22:03:22.362408-0800 Benchmark[554:39729] Invalid group id in source layers: test_layer
2025-01-22 22:03:22.362489-0800 Benchmark[554:39729] Validation failure: Tensor dimensions N1D1C768H1W32000 are not within supported range, N[1-65536]D[1-16384]C[1-65536]H[1-16384]W[1-16384].
loc("mps_transpose_84"("(mpsFileLoc): /Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":206:0)): error: 'anec.transpose' op Invalid configuration for the following reasons: Tensor dimensions N1D1C768H1W32000 are not within supported range, N[1-65536]D[1-16384]C[1-65536]H[1-16384]W[1-16384].
2025-01-22 22:03:22.581557-0800 Benchmark[554:39729] Invalid group id in source layers: test_layer
2025-01-22 22:03:22.581657-0800 Benchmark[554:39729] Validation failure: Tensor dimensions N1D1C768H1W32000 are not within supported range, N[1-65536]D[1-16384]C[1-65536]H[1-16384]W[1-16384].
loc("mps_matmul_108"("(mpsFileLoc): /Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":39:0)): error: 'anec.matmul' op Invalid configuration for the following reasons: Tensor dimensions N1D1C768H1W32000 are not within supported range, N[1-65536]D[1-16384]C[1-65536]H[1-16384]W[1-16384].
loc("mps_broadcast_to"("(mpsFileLoc): /Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":1327:0)): error: 'anec.broadcast' op failed: input cannot be broadcasted to the target shape
loc("mps_broadcast_to_2"("(mpsFileLoc): /Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":1327:0)): error: 'anec.broadcast' op failed: input cannot be broadcasted to the target shape
loc("mps_broadcast_to_4"("(mpsFileLoc): /Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":1327:0)): error: 'anec.broadcast' op failed: input cannot be broadcasted to the target shape
loc("mps_broadcast_to_6"("(mpsFileLoc): /Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":1327:0)): error: 'anec.broadcast' op failed: input cannot be broadcasted to the target shape
loc("mps_broadcast_to_8"("(mpsFileLoc): /Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":1327:0)): error: 'anec.broadcast' op failed: input cannot be broadcasted to the target shape
loc("mps_broadcast_to_10"("(mpsFileLoc): /Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":1327:0)): error: 'anec.broadcast' op failed: input cannot be broadcasted to the target shape
loc("mps_broadcast_to_12"("(mpsFileLoc): /Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":1327:0)): error: 'anec.broadcast' op failed: input cannot be broadcasted to the target shape
Job link: https://github.com/pytorch/executorch/actions/runs/12920386457/job/36038667522#step:14:2642
Versions
trunk