Skip to content

Reorg Failed Test Replay section [DOCS-11470] #30479

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
+29 −29
Open

Reorg Failed Test Replay section [DOCS-11470] #30479

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1c3ef98
Move "Failed Test Replay" section outside tabs
joepeeples Jul 15, 2025
e612481
Merge branch 'master' into joepeeples/11470-failed-test-replay-reorg
joepeeples Jul 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 29 additions & 29 deletions content/en/tests/flaky_tests/auto_test_retries.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@
[3]: /tests/setup/java/#compatibility
{{% /tab %}}

{{% tab "Javascript" %}}
{{% tab "JavaScript" %}}

### Compatibility

Expand All @@ -66,35 +66,7 @@
* `DD_CIVISIBILITY_FLAKY_RETRY_ENABLED` - set to 0 or false to explicitly disable retries even if the remote setting is enabled (default: true).
* `DD_CIVISIBILITY_FLAKY_RETRY_COUNT` - a non-negative number to change the maximum number of retries per test case (default: 5).

### Failed Test Replay

In addition to automatically retrying failed tests, Failed Test Replay allows you to see local variable data in the topmost frame of the test error's stack trace. Enable this feature with the **Failed Test Replay** toggle.

#### Create a logs index

Failed Test Replay creates logs that are sent to Datadog and appear alongside your regular application logs.

If you use [Exclusion filters][4], ensure Failed Test Replay logs are not filtered:

1. Create a logs index and [configure it][5] to the desired retention with **no sampling**.
2. Set the filter to match on the `source:dd_debugger` tag. All Failed Test Replay logs have this source.
3. Ensure that the new index takes precedence over any other index with filters that match that tag, because the first match wins.


After you enable this feature, you can see local variable data in failed tests:

{{< img src="continuous_integration/failed_test_replay_local_variables.png" alt="Failed Test Replay." style="width:100%" >}}


#### Known limitations

[jest-image-snapshot][2] is incompatible with `jest.retryTimes` unless `customSnapshotIdentifier` is passed (see [jest-image-snapshot docs][3]) to `toMatchImageSnapshot`. Therefore, auto test retries do not work unless `customSnapshotIdentifier` is used.

[1]: https://app.datadoghq.com/ci/settings/test-optimization
[2]: https://www.npmjs.com/package/jest-image-snapshot
[3]: https://github.com/americanexpress/jest-image-snapshot?tab=readme-ov-file#jestretrytimes
[4]: /logs/log_configuration/indexes/#exclusion-filters
[5]: /logs/log_configuration/indexes/#add-indexes

{{% /tab %}}

Expand Down Expand Up @@ -216,6 +188,30 @@

{{< /tabs >}}

### Failed Test Replay

Check warning on line 191 in content/en/tests/flaky_tests/auto_test_retries.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.headings 

'Failed Test Replay' should use sentence-style capitalization.

<div class="alert alert-info">Failed Test Replay is only supported for JavaScript.</a></div>

In addition to automatically retrying failed tests, Failed Test Replay allows you to see local variable data in the topmost frame of the test error's stack trace. Enable this feature with the **Failed Test Replay** toggle.

Check notice on line 195 in content/en/tests/flaky_tests/auto_test_retries.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.sentencelength 

Suggestion: Try to keep your sentence length to 25 words or fewer.

#### Create a logs index

Failed Test Replay creates logs that are sent to Datadog and appear alongside your regular application logs.

If you use [Exclusion filters][5], ensure Failed Test Replay logs are not filtered:

1. Create a logs index and [configure it][6] to the desired retention with **no sampling**.
2. Set the filter to match on the `source:dd_debugger` tag. All Failed Test Replay logs have this source.
3. Ensure that the new index takes precedence over any other index with filters that match that tag, because the first match wins.

After you enable this feature, you can see local variable data in failed tests:

{{< img src="continuous_integration/failed_test_replay_local_variables.png" alt="Failed Test Replay." style="width:100%" >}}

#### Known limitations

[jest-image-snapshot][7] is incompatible with `jest.retryTimes` unless `customSnapshotIdentifier` is passed (see [jest-image-snapshot docs][8]) to `toMatchImageSnapshot`. Therefore, auto test retries do not work unless `customSnapshotIdentifier` is used.

## Explore results in the Test Optimization Explorer

You can query the retried tests in the [Test Optimization Explorer][2]: they have the `@test.is_retry` tag set to `true` (some of them may also have the `@test.is_new` set to `true`, which indicates they have been retried by the [Early Flakiness Detection][3] feature).
Expand All @@ -232,3 +228,7 @@
[2]: /tests/explorer/
[3]: /tests/flaky_test_management/early_flake_detection
[4]: https://app.datadoghq.com/ci/settings/test-optimization
[5]: /logs/log_configuration/indexes/#exclusion-filters
[6]: /logs/log_configuration/indexes/#add-indexes
[7]: https://www.npmjs.com/package/jest-image-snapshot
[8]: https://github.com/americanexpress/jest-image-snapshot?tab=readme-ov-file#jestretrytimes
Loading