Regression Tests should check average/geomean QoR metrics

[kmurray]

Proposed Behaviour

After running regression tests we should automatically check the geomean of various run-time & QoR metrics to verify they are within the expected valid ranges.

Current Behaviour

Currently the regression test scripts check QoR ranges on a per-benchmark basis.
Since all of these metrics have some noise which can cause spurious failures the per-benchmark ranges are set relatively loose.

Possible Solution

We should extend parse_vtr_task.pl to produce geomean metric values (in addition to the per-benchmark results generated in parse_results.txt).

It should also be extended to then check that these geomean values are within the expected ranges and cause an error if not.

Currently parse_vtr_task.pl produces a file called qor_results.txt its not clear what this does and how it differs from the parse_results.txt or whether it could be used/changed to support this feature. If not clear why its there it could be removed.

Context

We should perform more stringent automated QoR checking to catch potential errors/bugs/degredations sooner.

A related issue is #371.

Loosely related #545.

[kmurray]

@vaughnbetz @renrichard FYI

[vaughnbetz]

Hi Richard,

When your current tasks are done, here's another useful upgrade.

[renrichard]

Sounds good, I have already included this and #535 in my to-do list as well.

[github-actions]

This issue has been inactive for a year and has been marked as stale. It will be closed in 15 days if it continues to be stale. If you believe this is still an issue, please add a comment.

[github-actions]

This issue has been marked stale for 15 days and has been automatically closed.

[vaughnbetz]

This is a good idea. @AmirhosseinPoolad : let's discuss in our next meeting and see if you can take on this upgrade.