ci: add pre-release performance quality gates prototype

[igoragoli]

https://datadoghq.atlassian.net/browse/APMSP-2000

Overview

Add a prototype for pre-release performance quality gates. This prototype doesn't actually gate anything.

Motivation

Necessary to validate the approach proposed in the pre-release performance quality gates RFC.

Testing

Testing pipelines:

• https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-py/-/pipelines/66090033 (containing extra testing jobs)
• https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-py/-/pipelines/66116717 (most recent one)

Threshold configuration files added in: https://github.com/DataDog/benchmarking-platform/pull/157

Notifications are correctly being sent to #pre-release-gates-prototype.

Checklist

• PR author has checked that all the criteria below are met
• The PR description includes an overview of the change
• The PR description articulates the motivation for the change
• The change includes tests OR the PR description describes a testing strategy
• The PR description notes risks associated with the change, if any
• Newly-added code is easy to change
• The change follows the library release note guidelines
• The change includes or references documentation updates if necessary
• Backport labels are set (if applicable)

Reviewer Checklist

• Reviewer has checked that all the criteria below are met
• Title is accurate
• All changes are related to the pull request's stated goal
• Avoids breaking API changes
• Testing strategy adequately addresses listed risks
• Newly-added code is easy to change
• Release note makes sense to a user of the library
• If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
• Backport labels are set in a manner that is consistent with the release branch maintenance policy

[github-actions]

CODEOWNERS have been resolved as:

.gitlab/benchmarks/macrobenchmarks.yml                                  @DataDog/python-guild @DataDog/apm-core-python

[github-actions]

Bootstrap import analysis

Comparison of import times between this PR and base.

Summary

The average import time from this PR is: 270 ± 10 ms.

The average import time from base is: 270 ± 10 ms.

The import time difference between this PR and base is: -6.1 ± 0.5 ms.

Import time breakdown

The following import paths have shrunk:

ddtrace.auto 1.549 ms (0.58%)

ddtrace.bootstrap.sitecustomize 0.855 ms (0.32%)

ddtrace.bootstrap.preload 0.855 ms (0.32%)

ddtrace.internal.remoteconfig.client 0.855 ms (0.32%)

ddtrace 0.694 ms (0.26%)

[pr-commenter]

Benchmarks

Benchmark execution time: 2025-05-29 19:10:39

Comparing candidate commit 02f0eaa in PR branch igoragoli/add-pre-release-gates with baseline commit 520cbf5 in branch main.

Found 1 performance improvements and 1 performance regressions! Performance is the same for 506 metrics, 8 unstable metrics.

scenario:iastdjangostartup-iast

• 🟩 execution_time [-438.653ms; -215.568ms] or [-18.403%; -9.044%]

scenario:telemetryaddmetric-1-gauge-metric-1-times

• 🟥 execution_time [+157.983ns; +236.050ns] or [+7.364%; +11.002%]

[erikayasuda]

LGTM, just one NB question 👍

[erikayasuda]

NB: I saw in the last pipeline that the macrobenchmarks pipeline was triggered and the SLO warning job failed. Initially I thought that we should be setting this as allow_fail: true so that a warning doesn't become a blocker for PRs... but then I realized that would make it silent on the main branch, which is what we're trying to alleviate 😅 If the intention of these checks is only for pre-release gating, maybe we can make this only trigger on main and release branches?

What are your thoughts? 🤔

Suggested change

	check-warning-breaches:
	extends: .check-threshold-breaches
	script:
	- cd platform && (git init && git remote add origin https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.ddbuild.io/DataDog/benchmarking-platform && git pull origin python/macrobenchmarks)
	- bp-runner bp-runner.fail-on-breach.warning.yml
	check-warning-breaches:
	extends: .check-threshold-breaches
	only:
	- main
	- /^v[0-9]+\.[0-9]+\.[0-9]+$/
	script:
	- cd platform && (git init && git remote add origin https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.ddbuild.io/DataDog/benchmarking-platform && git pull origin python/macrobenchmarks)
	- bp-runner bp-runner.fail-on-breach.warning.yml

[igoragoli]

Hi!

so that a warning doesn't become a blocker for PRs

I've made some updates. In summary, only one check-slo-breaches job is necessary, and it'll check for warnings with a range defined on the thresholds file (bp-runner.fail-on-breach.yml). Warnings won't fail the job 👍

we can make this only trigger on main and release branches?

Exactly, benchmarks considered for gating releases should be run on main and release branches.

This is not the case for Python macrobenchmarks, which only run on schedule: https://github.com/DataDog/dd-trace-py/blob/main/.gitlab-ci.yml#L91-L94

[brettlangdon]

something we found, the main trigger job needs to be strategy: depend for the trigger

[brettlangdon]

we might want to try and do similar as the microbenchmarks which will use the artifacts from the download_ddtrace_artifacts and use the pre-built wheels instead of installing from source which takes a few minutes.

[igoragoli]

Closing this due to updates on how breaches are checked, and due to the example PR I've added: #13584

@erikayasuda, this example PR can be a basis for the PR you need to add for https://datadoghq.atlassian.net/browse/APMSP-2051.