feat(junit): add includeRetries option to emit each retry as separate testcase

[vazidmansuri005]

Summary

Adds an includeRetries option to the JUnit reporter that emits retry information using the Maven Surefire JUnit XML schema:

• Flaky tests (eventually pass after retries): <flakyFailure> elements with <stackTrace>, no <failure> — test counts as passed
• Permanent failures (fail all retries): <failure> for the first attempt + <rerunFailure> elements for subsequent retries — test counts as failed

This follows the established schema from maven-surefire-plugin and is natively supported by Jenkins (JUnit plugin, since late 2025), Allure, and Gradle (mergeReruns). Tools that don't recognize these elements (GitHub Actions, GitLab CI, etc.) safely ignore them, so flaky tests still show as passed.

Configuration

// playwright.config.ts
export default {
  retries: 2,
  reporter: [['junit', { includeRetries: true }]],
};

Or via environment variable:

PLAYWRIGHT_JUNIT_INCLUDE_RETRIES=1

XML Output Examples

Flaky test (fails twice, passes on retry #2):

<testcase name="my test" classname="a.test.js" time="0.5">
  <flakyFailure message="expected true" type="expect.toBe">
    <stackTrace>Error: expect(received).toBe(expected)...</stackTrace>
  </flakyFailure>
  <flakyFailure message="expected true" type="expect.toBe">
    <stackTrace>Error: expect(received).toBe(expected)...</stackTrace>
  </flakyFailure>
</testcase>
<!-- tests="1" failures="0" — flaky test counts as passed -->

Permanent failure (fails all retries):

<testcase name="my test" classname="a.test.js" time="1.2">
  <failure message="expected true" type="expect.toBe">full formatted failure</failure>
  <rerunFailure message="expected true" type="expect.toBe">
    <stackTrace>Error: expect(received).toBe(expected)...</stackTrace>
  </rerunFailure>
</testcase>
<!-- tests="1" failures="1" — permanent failure counted once -->

Edge Cases

Scenario	Behavior
Flaky test (fails, fails, passes)	1 testcase, 0 failures, 2 <flakyFailure> elements
Permanent failure (fails all retries)	1 testcase, 1 failure, N-1 <rerunFailure> elements
No retries configured	Standard single testcase (unchanged)
includeRetries disabled (default)	Standard behavior (unchanged)
Thrown error (not assertion)	Uses <flakyError> / <rerunError> instead

Test Plan

• Flaky test emits <flakyFailure> elements, counts as passed
• Permanent failure emits <failure> + <rerunFailure>, counts as failed
• Default behavior unchanged when includeRetries is not set
• Works with both direct and merged (blob) reports
• All existing JUnit reporter tests pass

Closes #29446

[vazidmansuri005]

@microsoft-github-policy-service agree

[vazidmansuri005]

@yury-s @dgozman — Would appreciate your review on this. This adds an includeRetries option to the JUnit reporter (requested in #29446) so that each retry attempt is emitted as a separate <testcase>, enabling flaky test detection tools to see the full retry history. Default behavior is unchanged.

[dgozman]

@vazidmansuri005 Thank you for the PR!

I see that you chose to turn each retry into a <testcase>. I've also found this example that instead uses flakyFailure and rerunFailure. What is your intended way to consume this information from the report? Are there any existing junit report consumers out there that understand flakiness and show it nicely in some UI?

I'd like this feature to be immediately useful with existing integrations. If one would need a custom parser for such a junit report, they might as well use the json report that has more information already.

[vazidmansuri005]

@dgozman Great point — I've refactored the implementation to use the Maven Surefire JUnit XML schema with <flakyFailure> and <rerunFailure> elements instead of separate <testcase> entries.

How it works now:

• Flaky tests (eventually pass): <flakyFailure> elements with <stackTrace>, no <failure> → test counts as passed
• Permanent failures: <failure> + <rerunFailure> elements → test counts as failed (once)

Honest assessment of current tool support:

Tool	Parses these elements?	Shows in UI?
Jenkins JUnit Plugin	Yes (since v1380, Nov 2024)	Data only — accessible via CaseResult.getFlakyFailures() API, no UI changes yet
Jenkins Flaky Test Handler	Yes	Yes — flaky badge, deflake button, history charts (separate plugin)
Gradle Develocity	Yes (via mergeReruns)	Yes — FLAKY outcome badge, trend dashboard
Allure Report	Own retry mechanism	Has retries tab + bomb icon, but driven by historyId, not JUnit XML elements
GitHub Actions, GitLab CI, Azure DevOps	No	No — but safely ignore unknown elements

The immediate value even without dedicated UI:

1. Backwards-compatible — tools that don't understand flakyFailure simply ignore it. Since flaky tests have no <failure> element, they correctly show as passed. Permanent failures still have <failure>, so nothing breaks.
2. Accurate test counting — without this, a test with retries shows tests=1 failures=0 and you lose all retry information. With this, the XML preserves each retry's stack trace and output.
3. Established schema — this is exactly what Maven Surefire and Gradle produce. It's the de facto standard for representing retries in JUnit XML.
4. Programmatic access — Jenkins exposes it via API, and any custom JUnit XML parser (like junitparser for Python) can read flakyFailure/rerunFailure elements.

That said, I understand your concern about immediate usefulness. If you feel this isn't compelling enough without broader UI support, I'm open to suggestions on a better approach. The json reporter does have more information, but the JUnit format is what most CI/CD pipelines consume for test result aggregation, so having retry data in there (even if not all tools render it) seems valuable.

[vazidmansuri005]

Hi @dgozman, the changes are pushed — extracted buildSurefireRetryEntry and _addFailureEntry helpers to eliminate the duplication across flaky/rerun/testcase paths. stdio (system-out/system-err) is now handled in the shared helper as well. Would appreciate another look when you get a chance. Thanks!

[dgozman]

@vazidmansuri005 Please fix the lint failures. See contributing guide for more details.

[vazidmansuri005]

@dgozman Fixed — added non-null assertions for entry.children in _addFailureEntry to resolve the two TS18048 errors. Lint passes cleanly now.

[github-actions]

Test results for "MCP"

6 failed
❌ [chrome] › mcp/cli-session.spec.ts:127 › session reopen with different config @mcp-macos-15
❌ [chromium] › mcp/cli-session.spec.ts:127 › session reopen with different config @mcp-macos-15
❌ [chromium] › mcp/sse.spec.ts:158 › sse transport browser lifecycle (persistent) @mcp-macos-15
❌ [firefox] › mcp/cli-session.spec.ts:127 › session reopen with different config @mcp-macos-15
❌ [webkit] › mcp/cli-session.spec.ts:127 › session reopen with different config @mcp-macos-15
❌ [webkit] › mcp/http.spec.ts:207 › http transport browser lifecycle (persistent) @mcp-macos-15

5121 passed, 164 skipped

---

Merge workflow run.

[github-actions]

Test results for "tests 1"

1 flaky

⚠️ [firefox-library] › library/inspector/cli-codegen-1.spec.ts:1080 › cli codegen › should not throw csp directive violation errors `@firefox-ubuntu-22.04-node20`

38837 passed, 841 skipped

---

Merge workflow run.