-
Notifications
You must be signed in to change notification settings - Fork 668
Open
Labels
backend testerThis bug was found by the backend test suite.This bug was found by the backend test suite.
Description
Summary
This is the top-level tracking task for work regarding the ExecuTorch Test Infrastructure (Backend Tester, name TBD).
Tasks
- Split model test jobs into one job per model.
- Expose error messages in markdown summary.
- Clean up remaining operators tests failing for reasons outside of backend control or reasonable expectations.
- Move each backend test job into a dedicated workflow. This will make it easier to find specific results.
- Run a small smoke test on the test infrastructure on pull. This will prevent breakages.
- Look into displaying latest summary results on the HUD or other location.
- Investigate using pytest-xdist to recover from native crashes. They currently end the entire test run due to the unrecoverable nature of native crashes / memory corruption.
- Create a summary job to accumulate results for each backend and display top-level metrics (pass percentage for f32 models, operator pass percent).
Metadata
Metadata
Assignees
Labels
backend testerThis bug was found by the backend test suite.This bug was found by the backend test suite.
Type
Projects
Status
In progress