Skip to content

Conversation

strtgbb
Copy link
Collaborator

@strtgbb strtgbb commented Sep 2, 2025

Changelog category (leave one):

  • CI Fix or Improvement (changelog entry is not required)

Add a measure of test instability to the workflow report to help triage test failures.

Instability is broken down by check_name and test_name as some tests are only unstable under sanitizers.
Only the last 4 weeks of data is looked at to allow the metric to decay once a test is fixed.

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Performance tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All with Aarch64
  • All Regression
  • Disable CI Cache

@strtgbb strtgbb added cicd Improvements and fixes to the CICD process antalya-25.6 antalya-25.6.5 labels Sep 2, 2025
Copy link

github-actions bot commented Sep 2, 2025

Workflow [PR], commit [ad3281d]

@strtgbb
Copy link
Collaborator Author

strtgbb commented Sep 2, 2025

Issues with the current implementation:

  • Adding tests changes what group they are in and thus check_name, this results in an inaccurate calculation.
  • It is slow, report generation time increased from 15-20s to 30-50s. Maybe it should be a CLI option.

I could track just test_name, this solves the changing check_name issue, but we lose info about the effects of sanitizers.
Alternatively, I could make the code smart enough to ignore the check_name group number, at the cost of complexity and runtime.

@strtgbb strtgbb marked this pull request as ready for review September 2, 2025 18:15
@strtgbb strtgbb added port-everywhere PRs that must be ported to every other version branch developer enhancement Features to support the development experience labels Sep 4, 2025
@strtgbb
Copy link
Collaborator Author

strtgbb commented Sep 8, 2025

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
antalya-25.6 cicd Improvements and fixes to the CICD process developer enhancement Features to support the development experience port-everywhere PRs that must be ported to every other version branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant