Skip to content

HDDS-14072. Show structured test summary in CI result#10857

Open
chihsuan wants to merge 6 commits into
apache:masterfrom
chihsuan:HDDS-14072
Open

HDDS-14072. Show structured test summary in CI result#10857
chihsuan wants to merge 6 commits into
apache:masterfrom
chihsuan:HDDS-14072

Conversation

@chihsuan

@chihsuan chihsuan commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This PR implements the structured CI test summaries described in HDDS-14072.

  • Add dev-support/ci/junit_summary.py, a stdlib-only parser for Maven Surefire and Failsafe XML reports.
  • Append a structured Markdown test summary to the GitHub Actions job summary.
  • Recognize Surefire <flakyFailure> and <flakyError> elements as tests that recovered on rerun.
  • Add a quarantined-test table for the test-flaky split.
  • Register the parser unit tests with the existing Bats check through dev-support/ci/junit_summary.bats.

The generated summary includes:

  • total test count and cumulative test time
  • passed, failed, flaky, and skipped counts
  • details for failed, flaky, and skipped tests
  • a link to the archived test reports

Report discovery handles both normal module report directories and failed reports moved under target/<check>/. Iteration directories are excluded to avoid counting repeated runs multiple times.

Operational behavior

The summary is informational and cannot fail an otherwise successful check:

  • malformed or unreadable reports are skipped
  • the script always exits with status 0
  • the workflow step also uses continue-on-error: true
  • jobs without JUnit XML produce no summary

The parser uses Python's standard xml.etree.ElementTree. The input is generated by Maven in the same CI job, so no additional XML dependency is needed.

Out of scope

  • deriving a quarantine table from @Flaky and @Unhealthy annotations
  • Kafka-style new-test detection
  • aggregating all splits into one PR-level report
  • enabling rerunFailingTestsCount outside the flaky split

What is the link to the Apache JIRA?

https://issues.apache.org/jira/browse/HDDS-14072

How was this patch tested?

  • python3 -m unittest test_junit_summary -v: 16 tests passed
  • rat.sh: passed
  • checkstyle.sh: passed
  • Local Surefire smoke test with TestOzoneConfiguration: rendered 48 passing tests
  • Full fork CI run: all checks passed
    • non-test jobs produced no output
    • integration summaries included pass and skipped-test details
    • the flaky split reported 162 passed and 5 flaky tests, confirming real Surefire rerun detection
    • zizmor passed for the workflow change
Screenshot 2026-07-24 at 2 14 57 PM Screenshot 2026-07-24 at 2 19 14 PM

Generated-by: Claude Code (Opus 4.8)

Copilot AI review requested due to automatic review settings July 24, 2026 06:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@chihsuan
chihsuan marked this pull request as ready for review July 24, 2026 06:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants