Qualcomm AI Engine Direct - backward compatibility CI

[DannyYuyang-quic]

Summary

• add README instructions for updating Story LLaMA artifacts
• add backward compatibility validation for Story LLaMA in CI pipeline

Test plan

./.ci/scripts/test_qnn_static_llama.sh

cc: @haowhsu-quic, @cccclai

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12748

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Unrelated Failure

As of commit d211129 with merge base b8fe100 ():

NEW FAILURE - The following job has failed:

• pull / test-llava-runner-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 7fe120cd684f0042e4c5638072a88294464193e9a84fd1f0ad83702c0ca51f56 /exec failed with exit code 139

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / test-phi-3-mini-runner-linux / linux-job (gh) (trunk failure)
  AttributeError: 'StaticCacheConfig' object has no attribute 'get'

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[DannyYuyang-quic]

I've re-uploaded the story.pte file (~2MB now). I think this PR should be good to go.
model source: https://huggingface.co/karpathy/tinyllamas/tree/main/stories260K

[DannyYuyang-quic]

@pytorchbot label "release notes: qualcomm"

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this in D78871122.

[cccclai]

Thank you for adding the BC test!

[cccclai]

Seems like there is still error

Traceback (most recent call last):
  File "/pytorch/executorch/backends/qualcomm/tests/test_qnn_delegate.py", line 4159, in test_llama_stories_260k
    self.fail(msg["Error"])
AssertionError: [Errno 2] No such file or directory: './outputs/inference_speed.txt'

[DannyYuyang-quic]

Seems like there is still error

Traceback (most recent call last):
  File "/pytorch/executorch/backends/qualcomm/tests/test_qnn_delegate.py", line 4159, in test_llama_stories_260k
    self.fail(msg["Error"])
AssertionError: [Errno 2] No such file or directory: './outputs/inference_speed.txt'

Thank you for taking the time to catch that! I’ll look into it and fix the error.

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this in D78871122.

[cccclai]

The CI signal seems good! Thank you. Can you also rebase?

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this in D78871122.