Skip to content

ci: add pre-release performance quality gates prototype #13506

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed

ci: add pre-release performance quality gates prototype #13506

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
5d4bce5
Add gating and notification jobs
igoragoli May 26, 2025
d6c223a
Remove testing jobs
igoragoli May 26, 2025
e4f537b
Switch benchmarking-platform branch to python/macrobenchmarks
igoragoli May 26, 2025
095c96b
Remove "needs:" from threshold breach checks
igoragoli May 26, 2025
67c747a
Run threshold breach checks independently of macrobenchmark failures
igoragoli May 26, 2025
02f0eaa
Merge branch 'main' into igoragoli/add-pre-release-gates
erikayasuda May 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions .gitlab/benchmarks/macrobenchmarks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
stages:
- test
- gate
- notify

variables:
MACROBENCHMARKS_CI_IMAGE: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/benchmarking-platform:dd-trace-py-macrobenchmarks
Expand Down Expand Up @@ -362,3 +364,56 @@ attack-appsec-enabled-iast-enabled-ep-enabled:
DD_APPSEC_RASP_ENABLED: "true"
IAST_WORST: "true"
DD_SERVICE: "ddtrace-mb-attack-appsec-enabled-iast-enabled-ep-enabled"

.check-threshold-breaches:
stage: gate
when: always
tags: ["arch:amd64"]
image: registry.ddbuild.io/images/benchmarking-platform-tools-ubuntu:latest
artifacts:
name: "artifacts"
when: always
paths:
- platform/artifacts/
expire_in: 3 months

check-warning-breaches:
extends: .check-threshold-breaches
script:
- cd platform && (git init && git remote add origin https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.ddbuild.io/DataDog/benchmarking-platform && git pull origin python/macrobenchmarks)
- bp-runner bp-runner.fail-on-breach.warning.yml
Comment on lines +380 to +384
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NB: I saw in the last pipeline that the macrobenchmarks pipeline was triggered and the SLO warning job failed. Initially I thought that we should be setting this as allow_fail: true so that a warning doesn't become a blocker for PRs... but then I realized that would make it silent on the main branch, which is what we're trying to alleviate 😅 If the intention of these checks is only for pre-release gating, maybe we can make this only trigger on main and release branches?

What are your thoughts? 🤔

Suggested change
check-warning-breaches:
extends: .check-threshold-breaches
script:
- cd platform && (git init && git remote add origin https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.ddbuild.io/DataDog/benchmarking-platform && git pull origin python/macrobenchmarks)
- bp-runner bp-runner.fail-on-breach.warning.yml
check-warning-breaches:
extends: .check-threshold-breaches
only:
- main
- /^v[0-9]+\.[0-9]+\.[0-9]+$/
script:
- cd platform && (git init && git remote add origin https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.ddbuild.io/DataDog/benchmarking-platform && git pull origin python/macrobenchmarks)
- bp-runner bp-runner.fail-on-breach.warning.yml

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi!

so that a warning doesn't become a blocker for PRs

I've made some updates. In summary, only one check-slo-breaches job is necessary, and it'll check for warnings with a range defined on the thresholds file (bp-runner.fail-on-breach.yml). Warnings won't fail the job 👍

we can make this only trigger on main and release branches?

Exactly, benchmarks considered for gating releases should be run on main and release branches.

This is not the case for Python macrobenchmarks, which only run on schedule: https://github.com/DataDog/dd-trace-py/blob/main/.gitlab-ci.yml#L91-L94


check-slo-breaches:
extends: .check-threshold-breaches
script:
- cd platform && (git init && git remote add origin https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.ddbuild.io/DataDog/benchmarking-platform && git pull origin python/macrobenchmarks)
- bp-runner bp-runner.fail-on-breach.slos.yml

# Taken from https://github.com/DataDog/slack-notifier/blob/master/gitlab-pipeline-templates/v3-sdm/template.yml
.slack-notifier-base:
stage: notify
image: registry.ddbuild.io/slack-notifier:v27936653-9a2a7db-sdm-gbi-jammy@sha256:c9d1145319d1904fa72ea97904a15200d3cb684324723f9e1700bc02cc85065c
tags: [ "arch:amd64" ]
before_script:
- if [[ $GITLAB_USER_LOGIN = "codesync" || $GITLAB_USER_LOGIN = "ddci-service-account" ]]; then EMAIL=$(git show -s --format="%ae" HEAD); else EMAIL=$GITLAB_USER_EMAIL; fi
- SLACK_AUTHOR=$(echo $EMAIL | email2slackid || echo "")
- if [ -z "$SLACK_AUTHOR" ]; then echo "author not found or unsubscribed"; fi
- BUILD_URL="$CI_PROJECT_URL/pipelines/$CI_PIPELINE_ID"

notify-slo-breaches:
extends: .slack-notifier-base
needs: ["check-slo-breaches"]
when: on_failure
script:
# Ideally add the SLACK_AUTHOR once dd-trace-py can access SDS: https://datadoghq.atlassian.net/wiki/spaces/SDA/pages/3175317505/Accessing+SDS+data+externally+via+Graphql
- 'MESSAGE_TEXT=":x: $CI_COMMIT_REF_NAME - Performance SLOs in $CI_PROJECT_NAME were breached in <$BUILD_URL|$CI_PIPELINE_ID>. Please look into it."'
- postmessage "pre-release-gate-prototype" "$MESSAGE_TEXT"

notify-warning-breaches:
extends: .slack-notifier-base
needs: ["check-warning-breaches"]
when: on_failure
script:
# Ideally add the SLACK_AUTHOR once dd-trace-py can access SDS: https://datadoghq.atlassian.net/wiki/spaces/SDA/pages/3175317505/Accessing+SDS+data+externally+via+Graphql
- 'MESSAGE_TEXT=":warning: $CI_COMMIT_REF_NAME - Performance SLOs in $CI_PROJECT_NAME are about to be breached. A warning threshold in <$BUILD_URL|$CI_PIPELINE_ID> was triggered. Please look into it."'
- postmessage "pre-release-gate-prototype" "$MESSAGE_TEXT"
Loading