Add code to Bundleio to generate error stats #12051
Conversation
Signed-off-by: Zingo Andersen <[email protected]> Change-Id: Ib51b22c80954c87812b81b6fa9798ace705a555a
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12051
Note: Links to docs will display an error until the docs builds have been completed. ❗ 1 Active SEVsThere are 1 currently active SEVs. If your PR is affected, please view them below: ✅ No FailuresAs of commit 49542cb with merge base 142b1c6 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
zingo
added
release notes: devtools
|
Hi @digantdesai and @mergennachin this PR will add a new method/API to BundleIO and maybe you want to involve the proper people about this :) Also is the metric sane? And the way to propagate it back to the runner? My basics intentions is to be able to log/track models over time in a better way then PASS/FAIL on a set atol/rtol, as it easily miss if we could have set rtol/atol lower when improving stuff. Something like this is also useful to get a good guess of what atol/rtol could be to make it work instead of a lot of trail and error with different values. |
| double abs_err = std::abs(a_data[k] - e_data[k]); | ||
| double relative_divider = | ||
| std::max(std::abs(a_data[k]), std::abs(e_data[k])); | ||
| relative_divider = std::max(relative_divider, eps); | ||
| double relative_err = abs_err / relative_divider; | ||
|
|
||
| sum_abs += abs_err; | ||
| max_abs = std::max(max_abs, abs_err); | ||
| sum_rel += relative_err; | ||
| max_rel = std::max(max_rel, relative_err); |
There was a problem hiding this comment.
Is this good? I'm no ML-math-stats person so if this can be improved we should in PR or after :)
Add a way to get error stats/metrics between actual and reference output.
cc @digantdesai @freddan80 @per @oscarandersson8218