Skip to content

refactor: implement native lock objects #14028

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Conversation

P403n1x87
Copy link
Contributor

@P403n1x87 P403n1x87 commented Jul 16, 2025

We implement native lock objects to avoid using the ones from the standard library. This is yet another step to isolate the library from interacting with potentially critical stdlib modules that could get patched by the target application (e.g. gevent).

Performance Analysis

The current implementation of forksafe locks is based on a proxy object wrapping of the Python locks from the standard library that adds about 10x overhead when acquiring and releasing a lock. On local tests based on acquiring and releasing a lock in a loop over 1M iterations, the following are the numbers obtained

Implementation Timing
Current 1.05s
This PR 0.14s
Python stdlib locks 0.09s

Checklist

  • PR author has checked that all the criteria below are met
  • The PR description includes an overview of the change
  • The PR description articulates the motivation for the change
  • The change includes tests OR the PR description describes a testing strategy
  • The PR description notes risks associated with the change, if any
  • Newly-added code is easy to change
  • The change follows the library release note guidelines
  • The change includes or references documentation updates if necessary
  • Backport labels are set (if applicable)

Reviewer Checklist

  • Reviewer has checked that all the criteria below are met
  • Title is accurate
  • All changes are related to the pull request's stated goal
  • Avoids breaking API changes
  • Testing strategy adequately addresses listed risks
  • Newly-added code is easy to change
  • Release note makes sense to a user of the library
  • If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
  • Backport labels are set in a manner that is consistent with the release branch maintenance policy

@P403n1x87 P403n1x87 added the changelog/no-changelog A changelog entry is not required for this PR. label Jul 16, 2025
Copy link
Contributor

github-actions bot commented Jul 16, 2025

CODEOWNERS have been resolved as:

ddtrace/internal/_threads/lock.hpp                                      @DataDog/apm-core-python
ddtrace/internal/threads.py                                             @DataDog/apm-core-python
ddtrace/_monkey.py                                                      @DataDog/apm-core-python
ddtrace/_trace/context.py                                               @DataDog/apm-sdk-api-python
ddtrace/_trace/processor/__init__.py                                    @DataDog/apm-sdk-api-python
ddtrace/_trace/tracer.py                                                @DataDog/apm-sdk-api-python
ddtrace/appsec/_iast/_overhead_control_engine.py                        @DataDog/asm-python
ddtrace/contrib/internal/subprocess/patch.py                            @DataDog/asm-python
ddtrace/debugging/_encoding.py                                          @DataDog/debugger-python
ddtrace/debugging/_probe/registry.py                                    @DataDog/debugger-python
ddtrace/internal/_encoding.pyx                                          @DataDog/apm-core-python
ddtrace/internal/_threads.cpp                                           @DataDog/apm-core-python
ddtrace/internal/_threads.pyi                                           @DataDog/apm-core-python
ddtrace/internal/ci_visibility/encoder.py                               @DataDog/ci-app-libraries
ddtrace/internal/datastreams/processor.py                               @DataDog/data-streams-monitoring
ddtrace/internal/datastreams/schemas/schema_sampler.py                  @DataDog/data-streams-monitoring
ddtrace/internal/forksafe.py                                            @DataDog/apm-core-python
ddtrace/internal/periodic.py                                            @DataDog/apm-core-python
ddtrace/internal/processor/endpoint_call_counter.py                     @DataDog/apm-core-python
ddtrace/internal/processor/stats.py                                     @DataDog/apm-core-python
ddtrace/internal/rate_limiter.py                                        @DataDog/apm-core-python
ddtrace/internal/runtime/runtime_metrics.py                             @DataDog/apm-sdk-api-python
ddtrace/internal/service.py                                             @DataDog/apm-core-python
ddtrace/internal/telemetry/metrics_namespaces.pyx                       @DataDog/apm-core-python
ddtrace/internal/utils/cache.py                                         @DataDog/apm-core-python
ddtrace/internal/writer/writer.py                                       @DataDog/apm-core-python
ddtrace/llmobs/_evaluators/runner.py                                    @DataDog/ml-observability
ddtrace/llmobs/_llmobs.py                                               @DataDog/ml-observability
ddtrace/llmobs/_log_writer.py                                           @DataDog/ml-observability
ddtrace/llmobs/_writer.py                                               @DataDog/ml-observability
ddtrace/opentracer/span.py                                              @DataDog/apm-sdk-api-python
tests/internal/test_forksafe.py                                         @DataDog/apm-core-python

@P403n1x87 P403n1x87 force-pushed the chore/native-lock branch from c9a1695 to ea0afb2 Compare July 16, 2025 15:35
Copy link
Contributor

github-actions bot commented Jul 16, 2025

Bootstrap import analysis

Comparison of import times between this PR and base.

Summary

The average import time from this PR is: 278 ± 4 ms.

The average import time from base is: 280 ± 4 ms.

The import time difference between this PR and base is: -2.4 ± 0.2 ms.

Import time breakdown

The following import paths have appeared:

ddtrace.auto 4.021 ms (1.45%)
ddtrace 4.021 ms (1.45%)
ddtrace._logger 4.021 ms (1.45%)
ddtrace.internal.telemetry 4.021 ms (1.45%)
ddtrace.settings._agent 4.021 ms (1.45%)
ddtrace.settings 4.021 ms (1.45%)
ddtrace.settings.http 4.021 ms (1.45%)
ddtrace.internal.utils.cache 4.021 ms (1.45%)
ddtrace.internal.threads 4.021 ms (1.45%)
ddtrace.internal.forksafe 2.721 ms (0.98%)
wrapt 2.465 ms (0.89%)
wrapt.__wrapt__ 0.908 ms (0.33%)
wrapt.wrappers 0.439 ms (0.16%)
wrapt._wrappers 0.279 ms (0.10%)
wrapt.importer 0.516 ms (0.19%)
importlib.util 0.307 ms (0.11%)
importlib._abc 0.196 ms (0.07%)
wrapt.decorators 0.506 ms (0.18%)
wrapt.arguments 0.129 ms (0.05%)
wrapt.patches 0.185 ms (0.07%)
wrapt.weakrefs 0.151 ms (0.05%)
ddtrace.internal._threads 1.132 ms (0.41%)

The following import paths have disappeared:

ddtrace.auto 3.926 ms (1.41%)
ddtrace 3.926 ms (1.41%)
ddtrace._logger 3.926 ms (1.41%)
ddtrace.internal.telemetry 3.926 ms (1.41%)
ddtrace.settings._agent 2.440 ms (0.88%)
ddtrace.settings 2.440 ms (0.88%)
ddtrace.settings.integration 2.440 ms (0.88%)
ddtrace.vendor.debtcollector 2.440 ms (0.88%)
ddtrace.vendor 2.440 ms (0.88%)
ddtrace.internal.module 2.440 ms (0.88%)
ddtrace.internal.wrapping.context 2.137 ms (0.77%)
ddtrace.internal.utils.inspection 2.137 ms (0.77%)
ddtrace.internal.safety 2.137 ms (0.77%)
wrapt 2.137 ms (0.77%)
wrapt.__wrapt__ 0.966 ms (0.35%)
wrapt.wrappers 0.498 ms (0.18%)
wrapt._wrappers 0.279 ms (0.10%)
wrapt.decorators 0.424 ms (0.15%)
wrapt.arguments 0.116 ms (0.04%)
wrapt.importer 0.213 ms (0.08%)
wrapt.patches 0.184 ms (0.07%)
wrapt.weakrefs 0.152 ms (0.05%)
importlib.util 0.302 ms (0.11%)
importlib._abc 0.184 ms (0.07%)
ddtrace.internal.telemetry.writer 1.486 ms (0.53%)
ddtrace.internal.periodic 1.133 ms (0.41%)
ddtrace.internal._threads 1.133 ms (0.41%)
ddtrace.internal.forksafe 0.354 ms (0.13%)

The following import paths have grown:

ddtrace.auto 1.406 ms (0.51%)
ddtrace 1.253 ms (0.45%)
ddtrace._logger 1.253 ms (0.45%)
ddtrace.internal.telemetry 1.253 ms (0.45%)
ddtrace.internal.telemetry.writer 0.957 ms (0.34%)
http.client 0.957 ms (0.34%)
email.parser 0.807 ms (0.29%)
email.feedparser 0.807 ms (0.29%)
email.errors 0.679 ms (0.24%)
email._policybase 0.127 ms (0.05%)
email.utils 0.127 ms (0.05%)
random 0.075 ms (0.03%)
bisect 0.075 ms (0.03%)
email._parseaddr 0.052 ms (0.02%)
calendar 0.052 ms (0.02%)
http 0.077 ms (0.03%)
email.message 0.073 ms (0.03%)
email._encoded_words 0.073 ms (0.03%)
ddtrace.settings._agent 0.296 ms (0.11%)
ddtrace.settings 0.296 ms (0.11%)
ddtrace.settings.integration 0.221 ms (0.08%)
ddtrace.vendor.debtcollector 0.221 ms (0.08%)
ddtrace.vendor 0.145 ms (0.05%)
ddtrace.internal.module 0.145 ms (0.05%)
ddtrace.internal.wrapping.context 0.069 ms (0.02%)
ddtrace.internal.utils.inspection 0.069 ms (0.02%)
ddtrace.vendor.debtcollector.moves 0.077 ms (0.03%)
ddtrace.settings.http 0.075 ms (0.03%)
ddtrace.internal.utils.http 0.075 ms (0.03%)
dataclasses 0.075 ms (0.03%)
ddtrace.bootstrap.sitecustomize 0.152 ms (0.05%)
ddtrace._trace.trace_handlers 0.081 ms (0.03%)
ddtrace.ext.db 0.081 ms (0.03%)
ddtrace.contrib.internal.subprocess.constants 0.071 ms (0.03%)

The following import paths have shrunk:

ddtrace.auto 3.640 ms (1.31%)
ddtrace 2.140 ms (0.77%)
ddtrace._logger 1.458 ms (0.52%)
ddtrace.internal.telemetry 1.458 ms (0.52%)
ddtrace.internal.telemetry.writer 0.926 ms (0.33%)
http.client 0.896 ms (0.32%)
email.parser 0.819 ms (0.29%)
email.feedparser 0.819 ms (0.29%)
email._policybase 0.819 ms (0.29%)
email.header 0.692 ms (0.25%)
email.charset 0.692 ms (0.25%)
email.utils 0.065 ms (0.02%)
datetime 0.065 ms (0.02%)
_datetime 0.065 ms (0.02%)
ddtrace.internal.periodic 0.031 ms (0.01%)
ddtrace.settings._agent 0.434 ms (0.16%)
ddtrace.settings 0.348 ms (0.12%)
ddtrace.settings.integration 0.265 ms (0.10%)
ddtrace.vendor.debtcollector 0.265 ms (0.10%)
ddtrace.vendor 0.186 ms (0.07%)
ddtrace.internal.module 0.186 ms (0.07%)
ddtrace.internal.wrapping.context 0.186 ms (0.07%)
ddtrace.internal.wrapping 0.090 ms (0.03%)
bytecode 0.090 ms (0.03%)
bytecode.bytecode 0.090 ms (0.03%)
ddtrace.vendor.debtcollector.removals 0.078 ms (0.03%)
ddtrace.settings.http 0.083 ms (0.03%)
ddtrace.internal.utils.http 0.083 ms (0.03%)
email.encoders 0.083 ms (0.03%)
base64 0.083 ms (0.03%)
socket 0.086 ms (0.03%)
array 0.086 ms (0.03%)
ddtrace.settings._otel_remapper 0.098 ms (0.04%)
ddtrace.internal._unpatched 0.031 ms (0.01%)
json 0.031 ms (0.01%)
json.decoder 0.031 ms (0.01%)
re 0.031 ms (0.01%)
enum 0.031 ms (0.01%)
types 0.031 ms (0.01%)
ddtrace.bootstrap.sitecustomize 1.500 ms (0.54%)
ddtrace.bootstrap.preload 1.432 ms (0.52%)
ddtrace.internal.remoteconfig.client 0.688 ms (0.25%)
ddtrace._trace.trace_handlers 0.067 ms (0.02%)
ddtrace.contrib.trace_utils 0.067 ms (0.02%)
ddtrace.contrib.internal.trace_utils 0.067 ms (0.02%)
ddtrace.contrib.internal.trace_utils_base 0.067 ms (0.02%)
ddtrace.ext.user 0.067 ms (0.02%)

@P403n1x87 P403n1x87 force-pushed the chore/native-lock branch 2 times, most recently from 642ce61 to 7191c66 Compare July 17, 2025 11:10
We implement native lock objects to avoid using the ones from the
standard library. This is yet another step to isolate the library from
interacting with potentially critical stdlib modules that could get
patched by the target application (e.g. gevent).
@P403n1x87 P403n1x87 force-pushed the chore/native-lock branch from 7191c66 to 4baebd8 Compare July 17, 2025 12:40
@pr-commenter
Copy link

pr-commenter bot commented Jul 17, 2025

Performance SLOs

Benchmark execution time: 2025-07-23 09:24:07

Comparing candidate commit e3e6599 in branch chore/native-lock with performance thresholds.

Performance Test Results

Legend: ✅ Pass | ⚠️ Unstable | ❌ Fail

⚠️ coreapiscenario - 12/12 (2 unstable)

⚠️ context_with_data_listeners

Metric Status Value SLO % Under SLO
execution_time ⚠️ 13.652µs < 20.000µs +31.7%
max_rss_usage 29.118MB < 31.000MB +6.1%

✅ context_with_data_no_listeners

Metric Status Value SLO % Under SLO
execution_time 3.688µs < 10.000µs +63.1%
max_rss_usage 29.098MB < 31.000MB +6.1%

⚠️ context_with_data_only_all_listeners

Metric Status Value SLO % Under SLO
execution_time ⚠️ 13.678µs < 20.000µs +31.6%
max_rss_usage 29.098MB < 31.000MB +6.1%

✅ get_item_exists

Metric Status Value SLO % Under SLO
execution_time 638.738ns < 10.000µs +93.6%
max_rss_usage 29.138MB < 31.000MB +6.0%

✅ get_item_missing

Metric Status Value SLO % Under SLO
execution_time 686.308ns < 10.000µs +93.1%
max_rss_usage 29.157MB < 31.000MB +5.9%

✅ set_item

Metric Status Value SLO % Under SLO
execution_time 24.651µs < 30.000µs +17.8%
max_rss_usage 29.138MB < 31.000MB +6.0%
djangosimple - 22/22

✅ appsec

Metric Status Value SLO % Under SLO
execution_time 21.438ms < 22.300ms +3.9%
max_rss_usage 64.114MB < 65.500MB +2.1%

✅ exception-replay-enabled

Metric Status Value SLO % Under SLO
execution_time 1.359ms < 1.450ms +6.3%
max_rss_usage 63.563MB < 65.500MB +3.0%

✅ iast

Metric Status Value SLO % Under SLO
execution_time 21.329ms < 22.250ms +4.1%
max_rss_usage 64.075MB < 65.500MB +2.2%

✅ profiler

Metric Status Value SLO % Under SLO
execution_time 15.816ms < 16.550ms +4.4%
max_rss_usage 51.040MB < 53.000MB +3.7%

✅ span-code-origin

Metric Status Value SLO % Under SLO
execution_time 27.158ms < 28.200ms +3.7%
max_rss_usage 66.190MB < 68.000MB +2.7%

✅ tracer

Metric Status Value SLO % Under SLO
execution_time 21.367ms < 22.700ms +5.9%
max_rss_usage 64.094MB < 65.500MB +2.1%

✅ tracer-and-profiler

Metric Status Value SLO % Under SLO
execution_time 23.885ms < 24.900ms +4.1%
max_rss_usage 65.097MB < 67.000MB +2.8%

✅ tracer-no-caches

Metric Status Value SLO % Under SLO
execution_time 18.998ms < 19.650ms +3.3%
max_rss_usage 63.721MB < 65.500MB +2.7%

✅ tracer-no-databases

Metric Status Value SLO % Under SLO
execution_time 19.298ms < 20.100ms +4.0%
max_rss_usage 63.740MB < 65.500MB +2.7%

✅ tracer-no-middleware

Metric Status Value SLO % Under SLO
execution_time 21.162ms < 22.500ms +5.9%
max_rss_usage 64.075MB < 65.500MB +2.2%

✅ tracer-no-templates

Metric Status Value SLO % Under SLO
execution_time 21.071ms < 22.250ms +5.3%
max_rss_usage 64.094MB < 65.500MB +2.1%
errortrackingdjangosimple - 6/6

✅ errortracking-enabled-all

Metric Status Value SLO % Under SLO
execution_time 18.605ms < 19.850ms +6.3%
max_rss_usage 63.977MB < 65.500MB +2.3%

✅ errortracking-enabled-user

Metric Status Value SLO % Under SLO
execution_time 18.938ms < 19.400ms +2.4%
max_rss_usage 64.114MB < 65.500MB +2.1%

✅ tracer-enabled

Metric Status Value SLO % Under SLO
execution_time 18.616ms < 19.450ms +4.3%
max_rss_usage 63.721MB < 65.500MB +2.7%
errortrackingflasksqli - 6/6

✅ errortracking-enabled-all

Metric Status Value SLO % Under SLO
execution_time 2.112ms < 2.300ms +8.2%
max_rss_usage 51.550MB < 53.000MB +2.7%

✅ errortracking-enabled-user

Metric Status Value SLO % Under SLO
execution_time 2.119ms < 2.250ms +5.8%
max_rss_usage 51.649MB < 53.000MB +2.5%

✅ tracer-enabled

Metric Status Value SLO % Under SLO
execution_time 2.111ms < 2.300ms +8.2%
max_rss_usage 51.295MB < 53.000MB +3.2%
flasksimple - 14/14

✅ appsec-get

Metric Status Value SLO % Under SLO
execution_time 4.607ms < 4.750ms +3.0%
max_rss_usage 63.190MB < 64.000MB +1.3%

✅ appsec-post

Metric Status Value SLO % Under SLO
execution_time 6.636ms < 6.750ms +1.7%
max_rss_usage 63.230MB < 64.000MB +1.2%

✅ appsec-telemetry

Metric Status Value SLO % Under SLO
execution_time 4.600ms < 4.750ms +3.2%
max_rss_usage 63.170MB < 64.000MB +1.3%

✅ debugger

Metric Status Value SLO % Under SLO
execution_time 1.856ms < 2.000ms +7.2%
max_rss_usage 41.981MB < 44.000MB +4.6%

✅ iast-get

Metric Status Value SLO % Under SLO
execution_time 1.859ms < 2.000ms +7.1%
max_rss_usage 44.712MB < 45.000MB +0.6%

✅ profiler

Metric Status Value SLO % Under SLO
execution_time 1.980ms < 2.100ms +5.7%
max_rss_usage 43.597MB < 44.000MB +0.9%

✅ tracer

Metric Status Value SLO % Under SLO
execution_time 3.388ms < 3.650ms +7.2%
max_rss_usage 51.710MB < 53.000MB +2.4%
flasksqli - 6/6

✅ appsec-enabled

Metric Status Value SLO % Under SLO
execution_time 3.954ms < 4.200ms +5.9%
max_rss_usage 63.484MB < 66.000MB +3.8%

✅ iast-enabled

Metric Status Value SLO % Under SLO
execution_time 2.550ms < 2.800ms +8.9%
max_rss_usage 56.623MB < 58.000MB +2.4%

✅ tracer-enabled

Metric Status Value SLO % Under SLO
execution_time 2.103ms < 2.250ms +6.6%
max_rss_usage 51.236MB < 53.000MB +3.3%
httppropagationextract - 60/60

✅ all_styles_all_headers

Metric Status Value SLO % Under SLO
execution_time 80.482µs < 100.000µs +19.5%
max_rss_usage 29.197MB < 31.000MB +5.8%

✅ b3_headers

Metric Status Value SLO % Under SLO
execution_time 13.823µs < 20.000µs +30.9%
max_rss_usage 29.197MB < 31.000MB +5.8%

✅ b3_single_headers

Metric Status Value SLO % Under SLO
execution_time 12.761µs < 20.000µs +36.2%
max_rss_usage 29.295MB < 31.000MB +5.5%

✅ datadog_tracecontext_tracestate_not_propagated_on_trace_id_no_match

Metric Status Value SLO % Under SLO
execution_time 63.743µs < 80.000µs +20.3%
max_rss_usage 29.236MB < 31.000MB +5.7%

✅ datadog_tracecontext_tracestate_propagated_on_trace_id_match

Metric Status Value SLO % Under SLO
execution_time 67.894µs < 80.000µs +15.1%
max_rss_usage 29.255MB < 31.000MB +5.6%

✅ empty_headers

Metric Status Value SLO % Under SLO
execution_time 1.398µs < 10.000µs +86.0%
max_rss_usage 29.255MB < 31.000MB +5.6%

✅ full_t_id_datadog_headers

Metric Status Value SLO % Under SLO
execution_time 22.797µs < 30.000µs +24.0%
max_rss_usage 29.138MB < 31.000MB +6.0%

✅ invalid_priority_header

Metric Status Value SLO % Under SLO
execution_time 6.335µs < 10.000µs +36.6%
max_rss_usage 29.295MB < 31.000MB +5.5%

✅ invalid_span_id_header

Metric Status Value SLO % Under SLO
execution_time 6.337µs < 10.000µs +36.6%
max_rss_usage 29.275MB < 31.000MB +5.6%

✅ invalid_tags_header

Metric Status Value SLO % Under SLO
execution_time 6.374µs < 10.000µs +36.3%
max_rss_usage 29.236MB < 31.000MB +5.7%

✅ invalid_trace_id_header

Metric Status Value SLO % Under SLO
execution_time 6.409µs < 10.000µs +35.9%
max_rss_usage 29.215MB < 31.000MB +5.8%

✅ large_header_no_matches

Metric Status Value SLO % Under SLO
execution_time 27.341µs < 30.000µs +8.9%
max_rss_usage 29.236MB < 31.000MB +5.7%

✅ large_valid_headers_all

Metric Status Value SLO % Under SLO
execution_time 28.597µs < 40.000µs +28.5%
max_rss_usage 29.255MB < 31.000MB +5.6%

✅ medium_header_no_matches

Metric Status Value SLO % Under SLO
execution_time 9.702µs < 20.000µs +51.5%
max_rss_usage 29.295MB < 31.000MB +5.5%

✅ medium_valid_headers_all

Metric Status Value SLO % Under SLO
execution_time 11.056µs < 20.000µs +44.7%
max_rss_usage 29.275MB < 31.000MB +5.6%

✅ none_propagation_style

Metric Status Value SLO % Under SLO
execution_time 1.496µs < 10.000µs +85.0%
max_rss_usage 29.255MB < 31.000MB +5.6%

✅ tracecontext_headers

Metric Status Value SLO % Under SLO
execution_time 33.445µs < 40.000µs +16.4%
max_rss_usage 29.314MB < 31.000MB +5.4%

✅ valid_headers_all

Metric Status Value SLO % Under SLO
execution_time 6.382µs < 10.000µs +36.2%
max_rss_usage 29.236MB < 31.000MB +5.7%

✅ valid_headers_basic

Metric Status Value SLO % Under SLO
execution_time 5.970µs < 10.000µs +40.3%
max_rss_usage 29.177MB < 31.000MB +5.9%

✅ wsgi_empty_headers

Metric Status Value SLO % Under SLO
execution_time 1.391µs < 10.000µs +86.1%
max_rss_usage 29.215MB < 31.000MB +5.8%

✅ wsgi_invalid_priority_header

Metric Status Value SLO % Under SLO
execution_time 6.396µs < 10.000µs +36.0%
max_rss_usage 29.255MB < 31.000MB +5.6%

✅ wsgi_invalid_span_id_header

Metric Status Value SLO % Under SLO
execution_time 1.404µs < 10.000µs +86.0%
max_rss_usage 29.236MB < 31.000MB +5.7%

✅ wsgi_invalid_tags_header

Metric Status Value SLO % Under SLO
execution_time 6.399µs < 10.000µs +36.0%
max_rss_usage 29.236MB < 31.000MB +5.7%

✅ wsgi_invalid_trace_id_header

Metric Status Value SLO % Under SLO
execution_time 6.421µs < 10.000µs +35.8%
max_rss_usage 29.215MB < 31.000MB +5.8%

✅ wsgi_large_header_no_matches

Metric Status Value SLO % Under SLO
execution_time 28.532µs < 40.000µs +28.7%
max_rss_usage 29.236MB < 31.000MB +5.7%

✅ wsgi_large_valid_headers_all

Metric Status Value SLO % Under SLO
execution_time 29.779µs < 40.000µs +25.6%
max_rss_usage 29.255MB < 31.000MB +5.6%

✅ wsgi_medium_header_no_matches

Metric Status Value SLO % Under SLO
execution_time 10.040µs < 20.000µs +49.8%
max_rss_usage 29.295MB < 31.000MB +5.5%

✅ wsgi_medium_valid_headers_all

Metric Status Value SLO % Under SLO
execution_time 11.504µs < 20.000µs +42.5%
max_rss_usage 29.197MB < 31.000MB +5.8%

✅ wsgi_valid_headers_all

Metric Status Value SLO % Under SLO
execution_time 6.353µs < 10.000µs +36.5%
max_rss_usage 29.236MB < 31.000MB +5.7%

✅ wsgi_valid_headers_basic

Metric Status Value SLO % Under SLO
execution_time 5.953µs < 10.000µs +40.5%
max_rss_usage 29.314MB < 31.000MB +5.4%
httppropagationinject - 16/16

✅ ids_only

Metric Status Value SLO % Under SLO
execution_time 18.953µs < 30.000µs +36.8%
max_rss_usage 29.255MB < 31.000MB +5.6%

✅ with_all

Metric Status Value SLO % Under SLO
execution_time 33.434µs < 40.000µs +16.4%
max_rss_usage 29.157MB < 31.000MB +5.9%

✅ with_dd_origin

Metric Status Value SLO % Under SLO
execution_time 26.124µs < 30.000µs +12.9%
max_rss_usage 29.236MB < 31.000MB +5.7%

✅ with_priority_and_origin

Metric Status Value SLO % Under SLO
execution_time 27.313µs < 40.000µs +31.7%
max_rss_usage 29.334MB < 31.000MB +5.4%

✅ with_sampling_priority

Metric Status Value SLO % Under SLO
execution_time 21.460µs < 30.000µs +28.5%
max_rss_usage 29.275MB < 31.000MB +5.6%

✅ with_tags

Metric Status Value SLO % Under SLO
execution_time 27.793µs < 40.000µs +30.5%
max_rss_usage 29.275MB < 31.000MB +5.6%

✅ with_tags_invalid

Metric Status Value SLO % Under SLO
execution_time 29.968µs < 40.000µs +25.1%
max_rss_usage 29.236MB < 31.000MB +5.7%

✅ with_tags_max_size

Metric Status Value SLO % Under SLO
execution_time 28.520µs < 40.000µs +28.7%
max_rss_usage 29.275MB < 31.000MB +5.6%
iast_aspects - 40/40

✅ re_expand_aspect

Metric Status Value SLO % Under SLO
execution_time 33.192µs < 40.000µs +17.0%
max_rss_usage 34.681MB < 35.000MB +0.9%

✅ re_expand_noaspect

Metric Status Value SLO % Under SLO
execution_time 28.709µs < 40.000µs +28.2%
max_rss_usage 34.623MB < 35.000MB +1.1%

✅ re_findall_aspect

Metric Status Value SLO % Under SLO
execution_time 3.697µs < 10.000µs +63.0%
max_rss_usage 34.701MB < 35.000MB +0.9%

✅ re_findall_noaspect

Metric Status Value SLO % Under SLO
execution_time 1.415µs < 10.000µs +85.8%
max_rss_usage 34.662MB < 35.000MB +1.0%

✅ re_finditer_aspect

Metric Status Value SLO % Under SLO
execution_time 5.120µs < 10.000µs +48.8%
max_rss_usage 34.623MB < 35.000MB +1.1%

✅ re_finditer_noaspect

Metric Status Value SLO % Under SLO
execution_time 1.407µs < 10.000µs +85.9%
max_rss_usage 34.721MB < 35.000MB +0.8%

✅ re_fullmatch_aspect

Metric Status Value SLO % Under SLO
execution_time 3.385µs < 10.000µs +66.2%
max_rss_usage 34.662MB < 35.000MB +1.0%

✅ re_fullmatch_noaspect

Metric Status Value SLO % Under SLO
execution_time 1.290µs < 10.000µs +87.1%
max_rss_usage 34.642MB < 35.000MB +1.0%

✅ re_group_aspect

Metric Status Value SLO % Under SLO
execution_time 3.471µs < 10.000µs +65.3%
max_rss_usage 34.681MB < 35.000MB +0.9%

✅ re_group_noaspect

Metric Status Value SLO % Under SLO
execution_time 1.595µs < 10.000µs +84.1%
max_rss_usage 34.642MB < 35.000MB +1.0%

✅ re_groups_aspect

Metric Status Value SLO % Under SLO
execution_time 3.566µs < 10.000µs +64.3%
max_rss_usage 34.681MB < 35.000MB +0.9%

✅ re_groups_noaspect

Metric Status Value SLO % Under SLO
execution_time 1.714µs < 10.000µs +82.9%
max_rss_usage 34.780MB < 35.000MB +0.6%

✅ re_match_aspect

Metric Status Value SLO % Under SLO
execution_time 3.412µs < 10.000µs +65.9%
max_rss_usage 34.721MB < 35.000MB +0.8%

✅ re_match_noaspect

Metric Status Value SLO % Under SLO
execution_time 1.284µs < 10.000µs +87.2%
max_rss_usage 34.681MB < 35.000MB +0.9%

✅ re_search_aspect

Metric Status Value SLO % Under SLO
execution_time 3.269µs < 10.000µs +67.3%
max_rss_usage 34.623MB < 35.000MB +1.1%

✅ re_search_noaspect

Metric Status Value SLO % Under SLO
execution_time 1.202µs < 10.000µs +88.0%
max_rss_usage 34.681MB < 35.000MB +0.9%

✅ re_sub_aspect

Metric Status Value SLO % Under SLO
execution_time 4.702µs < 10.000µs +53.0%
max_rss_usage 34.681MB < 35.000MB +0.9%

✅ re_sub_noaspect

Metric Status Value SLO % Under SLO
execution_time 1.516µs < 10.000µs +84.8%
max_rss_usage 34.701MB < 35.000MB +0.9%

✅ re_subn_aspect

Metric Status Value SLO % Under SLO
execution_time 4.915µs < 10.000µs +50.8%
max_rss_usage 34.780MB < 35.500MB +2.0%

✅ re_subn_noaspect

Metric Status Value SLO % Under SLO
execution_time 1.595µs < 10.000µs +84.0%
max_rss_usage 34.662MB < 35.000MB +1.0%
iastaspects - 118/118

✅ add_aspect

Metric Status Value SLO % Under SLO
execution_time 331.592ns < 10.000µs +96.7%
max_rss_usage 34.662MB < 35.000MB +1.0%

✅ add_inplace_aspect

Metric Status Value SLO % Under SLO
execution_time 327.685ns < 10.000µs +96.7%
max_rss_usage 34.681MB < 35.000MB +0.9%

✅ add_inplace_noaspect

Metric Status Value SLO % Under SLO
execution_time 314.644ns < 10.000µs +96.9%
max_rss_usage 34.681MB < 35.000MB +0.9%

✅ add_noaspect

Metric Status Value SLO % Under SLO
execution_time 275.542ns < 10.000µs +97.2%
max_rss_usage 34.662MB < 35.000MB +1.0%

✅ bytearray_aspect

Metric Status Value SLO % Under SLO
execution_time 1.846µs < 10.000µs +81.5%
max_rss_usage 34.681MB < 35.000MB +0.9%

✅ bytearray_extend_aspect

Metric Status Value SLO % Under SLO
execution_time 1.374µs < 10.000µs +86.3%
max_rss_usage 34.681MB < 35.000MB +0.9%

✅ bytearray_extend_noaspect

Metric Status Value SLO % Under SLO
execution_time 611.243ns < 10.000µs +93.9%
max_rss_usage 34.642MB < 35.000MB +1.0%

✅ bytearray_noaspect

Metric Status Value SLO % Under SLO
execution_time 477.342ns < 10.000µs +95.2%
max_rss_usage 34.642MB < 35.000MB +1.0%

✅ bytes_aspect

Metric Status Value SLO % Under SLO
execution_time 1.867µs < 10.000µs +81.3%
max_rss_usage 34.681MB < 35.000MB +0.9%

✅ bytes_noaspect

Metric Status Value SLO % Under SLO
execution_time 494.414ns < 10.000µs +95.1%
max_rss_usage 34.681MB < 35.000MB +0.9%

✅ bytesio_aspect

Metric Status Value SLO % Under SLO
execution_time 1.884µs < 10.000µs +81.2%
max_rss_usage 34.662MB < 35.000MB +1.0%

✅ bytesio_noaspect

Metric Status Value SLO % Under SLO
execution_time 494.492ns < 10.000µs +95.1%
max_rss_usage 34.740MB < 35.000MB +0.7%

✅ capitalize_aspect

Metric Status Value SLO % Under SLO
execution_time 737.375ns < 10.000µs +92.6%
max_rss_usage 34.623MB < 35.000MB +1.1%

✅ capitalize_noaspect

Metric Status Value SLO % Under SLO
execution_time 436.135ns < 10.000µs +95.6%
max_rss_usage 34.642MB < 35.000MB +1.0%

✅ casefold_aspect

Metric Status Value SLO % Under SLO
execution_time 730.947ns < 10.000µs +92.7%
max_rss_usage 34.623MB < 35.000MB +1.1%

✅ casefold_noaspect

Metric Status Value SLO % Under SLO
execution_time 368.899ns < 10.000µs +96.3%
max_rss_usage 34.681MB < 35.000MB +0.9%

✅ decode_aspect

Metric Status Value SLO % Under SLO
execution_time 726.084ns < 10.000µs +92.7%
max_rss_usage 34.740MB < 35.000MB +0.7%

✅ decode_noaspect

Metric Status Value SLO % Under SLO
execution_time 416.867ns < 10.000µs +95.8%
max_rss_usage 34.662MB < 35.000MB +1.0%

✅ encode_aspect

Metric Status Value SLO % Under SLO
execution_time 708.237ns < 10.000µs +92.9%
max_rss_usage 34.681MB < 35.000MB +0.9%

✅ encode_noaspect

Metric Status Value SLO % Under SLO
execution_time 403.151ns < 10.000µs +96.0%
max_rss_usage 34.701MB < 35.000MB +0.9%

✅ format_aspect

Metric Status Value SLO % Under SLO
execution_time 3.364µs < 10.000µs +66.4%
max_rss_usage 34.740MB < 35.000MB +0.7%

✅ format_map_aspect

Metric Status Value SLO % Under SLO
execution_time 3.127µs < 10.000µs +68.7%
max_rss_usage 34.721MB < 35.000MB +0.8%

✅ format_map_noaspect

Metric Status Value SLO % Under SLO
execution_time 774.721ns < 10.000µs +92.3%
max_rss_usage 34.681MB < 35.000MB +0.9%

✅ format_noaspect

Metric Status Value SLO % Under SLO
execution_time 590.733ns < 10.000µs +94.1%
max_rss_usage 34.740MB < 35.000MB +0.7%

✅ index_aspect

Metric Status Value SLO % Under SLO
execution_time 340.094ns < 10.000µs +96.6%
max_rss_usage 34.623MB < 35.000MB +1.1%

✅ index_noaspect

Metric Status Value SLO % Under SLO
execution_time 278.719ns < 10.000µs +97.2%
max_rss_usage 34.623MB < 35.000MB +1.1%

✅ join_aspect

Metric Status Value SLO % Under SLO
execution_time 1.216µs < 10.000µs +87.8%
max_rss_usage 34.623MB < 35.000MB +1.1%

✅ join_noaspect

Metric Status Value SLO % Under SLO
execution_time 490.735ns < 10.000µs +95.1%
max_rss_usage 34.662MB < 35.000MB +1.0%

✅ ljust_aspect

Metric Status Value SLO % Under SLO
execution_time 10.485µs < 20.000µs +47.6%
max_rss_usage 34.819MB < 35.500MB +1.9%

✅ ljust_noaspect

Metric Status Value SLO % Under SLO
execution_time 403.353ns < 10.000µs +96.0%
max_rss_usage 34.583MB < 35.500MB +2.6%

✅ lower_aspect

Metric Status Value SLO % Under SLO
execution_time 2.245µs < 10.000µs +77.5%
max_rss_usage 34.662MB < 35.500MB +2.4%

✅ lower_noaspect

Metric Status Value SLO % Under SLO
execution_time 367.036ns < 10.000µs +96.3%
max_rss_usage 34.603MB < 35.000MB +1.1%

✅ lstrip_aspect

Metric Status Value SLO % Under SLO
execution_time 11.009µs < 20.000µs +45.0%
max_rss_usage 34.858MB < 35.500MB +1.8%

✅ lstrip_noaspect

Metric Status Value SLO % Under SLO
execution_time 381.548ns < 10.000µs +96.2%
max_rss_usage 34.761MB < 35.000MB +0.7%

✅ modulo_aspect

Metric Status Value SLO % Under SLO
execution_time 581.344ns < 10.000µs +94.2%
max_rss_usage 34.662MB < 35.000MB +1.0%

✅ modulo_aspect_for_bytearray_bytearray

Metric Status Value SLO % Under SLO
execution_time 1.224µs < 10.000µs +87.8%
max_rss_usage 34.780MB < 35.000MB +0.6%

✅ modulo_aspect_for_bytes

Metric Status Value SLO % Under SLO
execution_time 746.040ns < 10.000µs +92.5%
max_rss_usage 34.603MB < 35.000MB +1.1%

✅ modulo_aspect_for_bytes_bytearray

Metric Status Value SLO % Under SLO
execution_time 956.814ns < 10.000µs +90.4%
max_rss_usage 34.701MB < 35.000MB +0.9%

✅ modulo_noaspect

Metric Status Value SLO % Under SLO
execution_time 627.649ns < 10.000µs +93.7%
max_rss_usage 34.642MB < 35.000MB +1.0%

✅ replace_aspect

Metric Status Value SLO % Under SLO
execution_time 4.606µs < 10.000µs +53.9%
max_rss_usage 34.642MB < 35.000MB +1.0%

✅ replace_noaspect

Metric Status Value SLO % Under SLO
execution_time 462.173ns < 10.000µs +95.4%
max_rss_usage 34.662MB < 35.500MB +2.4%

✅ repr_aspect

Metric Status Value SLO % Under SLO
execution_time 904.947ns < 10.000µs +91.0%
max_rss_usage 34.760MB < 35.000MB +0.7%

✅ repr_noaspect

Metric Status Value SLO % Under SLO
execution_time 414.224ns < 10.000µs +95.9%
max_rss_usage 34.681MB < 35.000MB +0.9%

✅ rstrip_aspect

Metric Status Value SLO % Under SLO
execution_time 10.989µs < 20.000µs +45.1%
max_rss_usage 34.800MB < 35.500MB +2.0%

✅ rstrip_noaspect

Metric Status Value SLO % Under SLO
execution_time 378.735ns < 10.000µs +96.2%
max_rss_usage 34.662MB < 35.000MB +1.0%

✅ slice_aspect

Metric Status Value SLO % Under SLO
execution_time 482.605ns < 10.000µs +95.2%
max_rss_usage 34.642MB < 35.000MB +1.0%

✅ slice_noaspect

Metric Status Value SLO % Under SLO
execution_time 443.147ns < 10.000µs +95.6%
max_rss_usage 34.642MB < 35.000MB +1.0%

✅ stringio_aspect

Metric Status Value SLO % Under SLO
execution_time 2.197µs < 10.000µs +78.0%
max_rss_usage 34.642MB < 35.000MB +1.0%

✅ stringio_noaspect

Metric Status Value SLO % Under SLO
execution_time 717.034ns < 10.000µs +92.8%
max_rss_usage 34.662MB < 35.000MB +1.0%

✅ strip_aspect

Metric Status Value SLO % Under SLO
execution_time 10.546µs < 20.000µs +47.3%
max_rss_usage 34.780MB < 35.500MB +2.0%

✅ strip_noaspect

Metric Status Value SLO % Under SLO
execution_time 381.935ns < 10.000µs +96.2%
max_rss_usage 34.721MB < 35.000MB +0.8%

✅ swapcase_aspect

Metric Status Value SLO % Under SLO
execution_time 2.434µs < 10.000µs +75.7%
max_rss_usage 34.662MB < 35.000MB +1.0%

✅ swapcase_noaspect

Metric Status Value SLO % Under SLO
execution_time 535.051ns < 10.000µs +94.6%
max_rss_usage 34.603MB < 35.000MB +1.1%

✅ title_aspect

Metric Status Value SLO % Under SLO
execution_time 2.373µs < 10.000µs +76.3%
max_rss_usage 34.662MB < 35.000MB +1.0%

✅ title_noaspect

Metric Status Value SLO % Under SLO
execution_time 506.051ns < 10.000µs +94.9%
max_rss_usage 34.701MB < 35.000MB +0.9%

✅ translate_aspect

Metric Status Value SLO % Under SLO
execution_time 3.221µs < 10.000µs +67.8%
max_rss_usage 34.681MB < 35.000MB +0.9%

✅ translate_noaspect

Metric Status Value SLO % Under SLO
execution_time 1.042µs < 10.000µs +89.6%
max_rss_usage 34.681MB < 35.000MB +0.9%

✅ upper_aspect

Metric Status Value SLO % Under SLO
execution_time 2.326µs < 10.000µs +76.7%
max_rss_usage 34.662MB < 35.000MB +1.0%

✅ upper_noaspect

Metric Status Value SLO % Under SLO
execution_time 364.870ns < 10.000µs +96.4%
max_rss_usage 34.662MB < 35.000MB +1.0%
iastaspectsospath - 24/24

✅ ospathbasename_aspect

Metric Status Value SLO % Under SLO
execution_time 4.039µs < 10.000µs +59.6%
max_rss_usage 34.701MB < 35.000MB +0.9%

✅ ospathbasename_noaspect

Metric Status Value SLO % Under SLO
execution_time 1.072µs < 10.000µs +89.3%
max_rss_usage 34.662MB < 35.000MB +1.0%

✅ ospathjoin_aspect

Metric Status Value SLO % Under SLO
execution_time 6.091µs < 10.000µs +39.1%
max_rss_usage 34.662MB < 35.000MB +1.0%

✅ ospathjoin_noaspect

Metric Status Value SLO % Under SLO
execution_time 2.289µs < 10.000µs +77.1%
max_rss_usage 34.721MB < 35.000MB +0.8%

✅ ospathnormcase_aspect

Metric Status Value SLO % Under SLO
execution_time 3.353µs < 10.000µs +66.5%
max_rss_usage 34.701MB < 35.000MB +0.9%

✅ ospathnormcase_noaspect

Metric Status Value SLO % Under SLO
execution_time 562.161ns < 10.000µs +94.4%
max_rss_usage 34.662MB < 35.000MB +1.0%

✅ ospathsplit_aspect

Metric Status Value SLO % Under SLO
execution_time 4.795µs < 10.000µs +52.0%
max_rss_usage 34.681MB < 35.000MB +0.9%

✅ ospathsplit_noaspect

Metric Status Value SLO % Under SLO
execution_time 1.577µs < 10.000µs +84.2%
max_rss_usage 34.662MB < 35.000MB +1.0%

✅ ospathsplitdrive_aspect

Metric Status Value SLO % Under SLO
execution_time 3.596µs < 10.000µs +64.0%
max_rss_usage 34.681MB < 35.000MB +0.9%

✅ ospathsplitdrive_noaspect

Metric Status Value SLO % Under SLO
execution_time 691.027ns < 10.000µs +93.1%
max_rss_usage 34.603MB < 35.000MB +1.1%

✅ ospathsplitext_aspect

Metric Status Value SLO % Under SLO
execution_time 5.131µs < 10.000µs +48.7%
max_rss_usage 34.681MB < 35.000MB +0.9%

✅ ospathsplitext_noaspect

Metric Status Value SLO % Under SLO
execution_time 1.382µs < 10.000µs +86.2%
max_rss_usage 34.662MB < 35.000MB +1.0%
iastaspectssplit - 12/12

✅ rsplit_aspect

Metric Status Value SLO % Under SLO
execution_time 1.540µs < 10.000µs +84.6%
max_rss_usage 34.681MB < 35.000MB +0.9%

✅ rsplit_noaspect

Metric Status Value SLO % Under SLO
execution_time 576.184ns < 10.000µs +94.2%
max_rss_usage 34.681MB < 35.000MB +0.9%

✅ split_aspect

Metric Status Value SLO % Under SLO
execution_time 1.504µs < 10.000µs +85.0%
max_rss_usage 34.623MB < 35.000MB +1.1%

✅ split_noaspect

Metric Status Value SLO % Under SLO
execution_time 568.702ns < 10.000µs +94.3%
max_rss_usage 34.701MB < 35.000MB +0.9%

✅ splitlines_aspect

Metric Status Value SLO % Under SLO
execution_time 1.455µs < 10.000µs +85.5%
max_rss_usage 34.721MB < 35.000MB +0.8%

✅ splitlines_noaspect

Metric Status Value SLO % Under SLO
execution_time 580.954ns < 10.000µs +94.2%
max_rss_usage 34.662MB < 35.000MB +1.0%
iastpropagation - 8/8

✅ no-propagation

Metric Status Value SLO % Under SLO
execution_time 49.100µs < 60.000µs +18.2%
max_rss_usage 34.229MB < 35.500MB +3.6%

✅ propagation_enabled

Metric Status Value SLO % Under SLO
execution_time 145.199µs < 160.000µs +9.3%
max_rss_usage 34.603MB < 35.500MB +2.5%

✅ propagation_enabled_100

Metric Status Value SLO % Under SLO
execution_time 1.567ms < 1.800ms +12.9%
max_rss_usage 34.642MB < 35.000MB +1.0%

✅ propagation_enabled_1000

Metric Status Value SLO % Under SLO
execution_time 28.872ms < 30.550ms +5.5%
max_rss_usage 34.681MB < 35.500MB +2.3%
otelsdkspan - 24/24

✅ add-event

Metric Status Value SLO % Under SLO
execution_time 40.612ms < 42.000ms +3.3%
max_rss_usage 31.595MB < 35.000MB +9.7%

✅ add-link

Metric Status Value SLO % Under SLO
execution_time 36.511ms < 38.550ms +5.3%
max_rss_usage 31.634MB < 35.000MB +9.6%

✅ add-metrics

Metric Status Value SLO % Under SLO
execution_time 221.189ms < 232.000ms +4.7%
max_rss_usage 31.517MB < 35.000MB +10.0%

✅ add-tags

Metric Status Value SLO % Under SLO
execution_time 210.534ms < 221.600ms +5.0%
max_rss_usage 31.517MB < 35.000MB +10.0%

✅ get-context

Metric Status Value SLO % Under SLO
execution_time 29.194ms < 31.300ms +6.7%
max_rss_usage 31.477MB < 35.000MB +10.1%

✅ is-recording

Metric Status Value SLO % Under SLO
execution_time 29.602ms < 31.000ms +4.5%
max_rss_usage 31.556MB < 35.000MB +9.8%

✅ record-exception

Metric Status Value SLO % Under SLO
execution_time 63.148ms < 65.850ms +4.1%
max_rss_usage 31.536MB < 35.000MB +9.9%

✅ set-status

Metric Status Value SLO % Under SLO
execution_time 32.370ms < 34.150ms +5.2%
max_rss_usage 31.575MB < 35.000MB +9.8%

✅ start

Metric Status Value SLO % Under SLO
execution_time 28.772ms < 30.150ms +4.6%
max_rss_usage 31.595MB < 35.000MB +9.7%

✅ start-finish

Metric Status Value SLO % Under SLO
execution_time 33.953ms < 35.350ms +4.0%
max_rss_usage 31.517MB < 35.000MB +10.0%

✅ start-finish-telemetry

Metric Status Value SLO % Under SLO
execution_time 34.188ms < 35.450ms +3.6%
max_rss_usage 31.575MB < 35.000MB +9.8%

✅ update-name

Metric Status Value SLO % Under SLO
execution_time 30.828ms < 33.400ms +7.7%
max_rss_usage 31.517MB < 35.000MB +10.0%
otelspan - 15/22

❌ add-event

Metric Status Value SLO % Under SLO
execution_time 44.078ms < 47.150ms +6.5%
max_rss_usage 43.749MB < 42.500MB -2.9%

✅ add-metrics

Metric Status Value SLO % Under SLO
execution_time 320.623ms < 344.800ms +7.0%
max_rss_usage 561.000MB < 562.000MB +0.2%

✅ add-tags

Metric Status Value SLO % Under SLO
execution_time 288.899ms < 314.000ms +8.0%
max_rss_usage 561.504MB < 563.500MB +0.4%

❌ get-context

Metric Status Value SLO % Under SLO
execution_time 85.165ms < 92.350ms +7.8%
max_rss_usage 46.162MB < 38.000MB -21.5%

❌ is-recording

Metric Status Value SLO % Under SLO
execution_time 41.718ms < 44.500ms +6.3%
max_rss_usage 42.672MB < 42.000MB -1.6%

❌ record-exception

Metric Status Value SLO % Under SLO
execution_time 60.412ms < 67.650ms +10.7%
max_rss_usage 43.971MB < 38.000MB -15.7%

❌ set-status

Metric Status Value SLO % Under SLO
execution_time 47.492ms < 50.400ms +5.8%
max_rss_usage 44.347MB < 42.000MB -5.6%

❌ start

Metric Status Value SLO % Under SLO
execution_time 40.928ms < 43.450ms +5.8%
max_rss_usage 44.114MB < 42.000MB -5.0%

✅ start-finish

Metric Status Value SLO % Under SLO
execution_time 80.986ms < 86.000ms +5.8%
max_rss_usage 31.791MB < 32.000MB +0.7%

✅ start-finish-telemetry

Metric Status Value SLO % Under SLO
execution_time 82.968ms < 86.000ms +3.5%
max_rss_usage 31.712MB < 32.000MB +0.9%

❌ update-name

Metric Status Value SLO % Under SLO
execution_time 42.942ms < 45.150ms +4.9%
max_rss_usage 43.504MB < 42.500MB -2.4%
packagespackageforrootmodulemapping - 4/4

✅ cache_off

Metric Status Value SLO % Under SLO
execution_time 344.938ms < 354.300ms +2.6%
max_rss_usage 35.406MB < 38.000MB +6.8%

✅ cache_on

Metric Status Value SLO % Under SLO
execution_time 384.638ns < 10.000µs +96.2%
max_rss_usage 33.450MB < 38.000MB +12.0%
packagesupdateimporteddependencies - 24/24

✅ import_many

Metric Status Value SLO % Under SLO
execution_time 155.475µs < 170.000µs +8.5%
max_rss_usage 33.933MB < 35.500MB +4.4%

✅ import_many_cached

Metric Status Value SLO % Under SLO
execution_time 121.135µs < 130.000µs +6.8%
max_rss_usage 34.039MB < 35.500MB +4.1%

✅ import_many_stdlib

Metric Status Value SLO % Under SLO
execution_time 1.611ms < 1.750ms +8.0%
max_rss_usage 34.197MB < 35.500MB +3.7%

✅ import_many_stdlib_cached

Metric Status Value SLO % Under SLO
execution_time 972.279µs < 1.100ms +11.6%
max_rss_usage 34.282MB < 35.500MB +3.4%

✅ import_many_unknown

Metric Status Value SLO % Under SLO
execution_time 836.439µs < 890.000µs +6.0%
max_rss_usage 34.168MB < 35.500MB +3.8%

✅ import_many_unknown_cached

Metric Status Value SLO % Under SLO
execution_time 788.313µs < 870.000µs +9.4%
max_rss_usage 34.066MB < 35.500MB +4.0%

✅ import_one

Metric Status Value SLO % Under SLO
execution_time 19.753µs < 30.000µs +34.2%
max_rss_usage 33.894MB < 35.500MB +4.5%

✅ import_one_cache

Metric Status Value SLO % Under SLO
execution_time 6.231µs < 10.000µs +37.7%
max_rss_usage 33.942MB < 35.500MB +4.4%

✅ import_one_stdlib

Metric Status Value SLO % Under SLO
execution_time 18.750µs < 20.000µs +6.2%
max_rss_usage 33.876MB < 35.500MB +4.6%

✅ import_one_stdlib_cache

Metric Status Value SLO % Under SLO
execution_time 6.282µs < 10.000µs +37.2%
max_rss_usage 33.931MB < 35.500MB +4.4%

✅ import_one_unknown

Metric Status Value SLO % Under SLO
execution_time 45.808µs < 50.000µs +8.4%
max_rss_usage 34.122MB < 35.500MB +3.9%

✅ import_one_unknown_cache

Metric Status Value SLO % Under SLO
execution_time 6.300µs < 10.000µs +37.0%
max_rss_usage 33.918MB < 35.500MB +4.5%
ratelimiter - 12/12

✅ defaults

Metric Status Value SLO % Under SLO
execution_time 2.386µs < 10.000µs +76.1%
max_rss_usage 28.764MB < 31.000MB +7.2%

✅ high_rate_limit

Metric Status Value SLO % Under SLO
execution_time 2.457µs < 10.000µs +75.4%
max_rss_usage 28.705MB < 31.000MB +7.4%

✅ long_window

Metric Status Value SLO % Under SLO
execution_time 2.391µs < 10.000µs +76.1%
max_rss_usage 28.705MB < 31.000MB +7.4%

✅ low_rate_limit

Metric Status Value SLO % Under SLO
execution_time 2.391µs < 10.000µs +76.1%
max_rss_usage 28.764MB < 31.000MB +7.2%

✅ no_rate_limit

Metric Status Value SLO % Under SLO
execution_time 824.191ns < 10.000µs +91.8%
max_rss_usage 28.705MB < 31.000MB +7.4%

✅ short_window

Metric Status Value SLO % Under SLO
execution_time 2.548µs < 10.000µs +74.5%
max_rss_usage 28.705MB < 31.000MB +7.4%
recursivecomputation - 8/8

✅ deep

Metric Status Value SLO % Under SLO
execution_time 307.947ms < 320.950ms +4.1%
max_rss_usage 29.884MB < 31.000MB +3.6%

✅ deep-profiled

Metric Status Value SLO % Under SLO
execution_time 346.062ms < 359.150ms +3.6%
max_rss_usage 34.583MB < 35.500MB +2.6%

✅ medium

Metric Status Value SLO % Under SLO
execution_time 6.989ms < 7.400ms +5.5%
max_rss_usage 29.098MB < 31.000MB +6.1%

✅ shallow

Metric Status Value SLO % Under SLO
execution_time 944.262µs < 1.050ms +10.1%
max_rss_usage 29.098MB < 31.000MB +6.1%
samplingrules - 8/8

✅ average_match

Metric Status Value SLO % Under SLO
execution_time 337.425µs < 350.000µs +3.6%
max_rss_usage 29.157MB < 31.000MB +5.9%

✅ high_match

Metric Status Value SLO % Under SLO
execution_time 508.895µs < 550.000µs +7.5%
max_rss_usage 29.098MB < 31.000MB +6.1%

✅ low_match

Metric Status Value SLO % Under SLO
execution_time 168.411µs < 190.000µs +11.4%
max_rss_usage 431.829MB < 432.500MB +0.2%

✅ very_low_match

Metric Status Value SLO % Under SLO
execution_time 8.666ms < 9.150ms +5.3%
max_rss_usage 54.865MB < 55.000MB +0.2%
sethttpmeta - 32/32

✅ all-disabled

Metric Status Value SLO % Under SLO
execution_time 12.352µs < 20.000µs +38.2%
max_rss_usage 29.628MB < 31.000MB +4.4%

✅ all-enabled

Metric Status Value SLO % Under SLO
execution_time 42.415µs < 50.000µs +15.2%
max_rss_usage 29.610MB < 31.000MB +4.5%

✅ collectipvariant_exists

Metric Status Value SLO % Under SLO
execution_time 42.927µs < 50.000µs +14.1%
max_rss_usage 29.669MB < 31.000MB +4.3%

✅ no-collectipvariant

Metric Status Value SLO % Under SLO
execution_time 42.530µs < 50.000µs +14.9%
max_rss_usage 29.610MB < 31.000MB +4.5%

✅ no-useragentvariant

Metric Status Value SLO % Under SLO
execution_time 40.981µs < 50.000µs +18.0%
max_rss_usage 29.727MB < 31.000MB +4.1%

✅ obfuscation-no-query

Metric Status Value SLO % Under SLO
execution_time 43.620µs < 50.000µs +12.8%
max_rss_usage 29.648MB < 31.000MB +4.4%

✅ obfuscation-regular-case-explicit-query

Metric Status Value SLO % Under SLO
execution_time 79.079µs < 90.000µs +12.1%
max_rss_usage 30.062MB < 31.000MB +3.0%

✅ obfuscation-regular-case-implicit-query

Metric Status Value SLO % Under SLO
execution_time 79.969µs < 90.000µs +11.1%
max_rss_usage 30.081MB < 31.000MB +3.0%

✅ obfuscation-send-querystring-disabled

Metric Status Value SLO % Under SLO
execution_time 157.091µs < 170.000µs +7.6%
max_rss_usage 30.101MB < 31.000MB +2.9%

✅ obfuscation-worst-case-explicit-query

Metric Status Value SLO % Under SLO
execution_time 150.742µs < 160.000µs +5.8%
max_rss_usage 30.062MB < 31.000MB +3.0%

✅ obfuscation-worst-case-implicit-query

Metric Status Value SLO % Under SLO
execution_time 157.486µs < 170.000µs +7.4%
max_rss_usage 30.081MB < 31.000MB +3.0%

✅ useragentvariant_exists_1

Metric Status Value SLO % Under SLO
execution_time 41.629µs < 50.000µs +16.7%
max_rss_usage 29.628MB < 31.000MB +4.4%

✅ useragentvariant_exists_2

Metric Status Value SLO % Under SLO
execution_time 42.845µs < 50.000µs +14.3%
max_rss_usage 29.688MB < 31.000MB +4.2%

✅ useragentvariant_exists_3

Metric Status Value SLO % Under SLO
execution_time 42.148µs < 50.000µs +15.7%
max_rss_usage 29.688MB < 31.000MB +4.2%

✅ useragentvariant_not_exists_1

Metric Status Value SLO % Under SLO
execution_time 41.760µs < 50.000µs +16.5%
max_rss_usage 29.707MB < 31.000MB +4.2%

✅ useragentvariant_not_exists_2

Metric Status Value SLO % Under SLO
execution_time 41.576µs < 50.000µs +16.8%
max_rss_usage 29.669MB < 31.000MB +4.3%
span - 26/26

✅ add-event

Metric Status Value SLO % Under SLO
execution_time 22.872ms < 26.200ms +12.7%
max_rss_usage 48.605MB < 49.000MB +0.8%

✅ add-metrics

Metric Status Value SLO % Under SLO
execution_time 91.145ms < 98.350ms +7.3%
max_rss_usage 626.139MB < 961.000MB +34.8%

✅ add-tags

Metric Status Value SLO % Under SLO
execution_time 149.276ms < 168.550ms +11.4%
max_rss_usage 626.917MB < 962.500MB +34.9%

✅ get-context

Metric Status Value SLO % Under SLO
execution_time 21.539ms < 23.700ms +9.1%
max_rss_usage 47.367MB < 47.500MB +0.3%

✅ is-recording

Metric Status Value SLO % Under SLO
execution_time 21.393ms < 23.900ms +10.5%
max_rss_usage 47.398MB < 47.500MB +0.2%

✅ record-exception

Metric Status Value SLO % Under SLO
execution_time 40.840ms < 44.500ms +8.2%
max_rss_usage 40.302MB < 40.500MB +0.5%

✅ set-status

Metric Status Value SLO % Under SLO
execution_time 23.277ms < 26.000ms +10.5%
max_rss_usage 47.357MB < 47.500MB +0.3%

✅ start

Metric Status Value SLO % Under SLO
execution_time 21.053ms < 23.500ms +10.4%
max_rss_usage 47.370MB < 47.500MB +0.3%

✅ start-finish

Metric Status Value SLO % Under SLO
execution_time 50.193ms < 52.500ms +4.4%
max_rss_usage 29.098MB < 31.000MB +6.1%

✅ start-finish-telemetry

Metric Status Value SLO % Under SLO
execution_time 51.409ms < 55.300ms +7.0%
max_rss_usage 29.098MB < 31.000MB +6.1%

✅ start-finish-traceid128

Metric Status Value SLO % Under SLO
execution_time 53.287ms < 56.050ms +4.9%
max_rss_usage 29.118MB < 31.000MB +6.1%

✅ start-traceid128

Metric Status Value SLO % Under SLO
execution_time 21.468ms < 24.600ms +12.7%
max_rss_usage 47.331MB < 47.500MB +0.4%

✅ update-name

Metric Status Value SLO % Under SLO
execution_time 22.160ms < 24.100ms +8.0%
max_rss_usage 48.005MB < 48.000MB -0.0%
telemetryaddmetric - 30/30

✅ 1-count-metric-1-times

Metric Status Value SLO % Under SLO
execution_time 2.259µs < 10.000µs +77.4%
max_rss_usage 29.137MB < 31.000MB +6.0%

✅ 1-count-metrics-100-times

Metric Status Value SLO % Under SLO
execution_time 166.021µs < 240.000µs +30.8%
max_rss_usage 29.118MB < 31.000MB +6.1%

✅ 1-distribution-metric-1-times

Metric Status Value SLO % Under SLO
execution_time 2.040µs < 10.000µs +79.6%
max_rss_usage 29.118MB < 31.000MB +6.1%

✅ 1-distribution-metrics-100-times

Metric Status Value SLO % Under SLO
execution_time 141.383µs < 210.000µs +32.7%
max_rss_usage 29.098MB < 31.000MB +6.1%

✅ 1-gauge-metric-1-times

Metric Status Value SLO % Under SLO
execution_time 1.821µs < 10.000µs +81.8%
max_rss_usage 29.118MB < 31.000MB +6.1%

✅ 1-gauge-metrics-100-times

Metric Status Value SLO % Under SLO
execution_time 123.631µs < 140.000µs +11.7%
max_rss_usage 29.098MB < 31.000MB +6.1%

✅ 1-rate-metric-1-times

Metric Status Value SLO % Under SLO
execution_time 2.272µs < 10.000µs +77.3%
max_rss_usage 29.137MB < 31.000MB +6.0%

✅ 1-rate-metrics-100-times

Metric Status Value SLO % Under SLO
execution_time 165.064µs < 230.000µs +28.2%
max_rss_usage 29.157MB < 31.000MB +5.9%

✅ 100-count-metrics-100-times

Metric Status Value SLO % Under SLO
execution_time 16.553ms < 22.500ms +26.4%
max_rss_usage 29.098MB < 31.000MB +6.1%

✅ 100-distribution-metrics-100-times

Metric Status Value SLO % Under SLO
execution_time 1.470ms < 2.100ms +30.0%
max_rss_usage 29.098MB < 31.000MB +6.1%

✅ 100-gauge-metrics-100-times

Metric Status Value SLO % Under SLO
execution_time 1.265ms < 1.400ms +9.6%
max_rss_usage 29.118MB < 31.000MB +6.1%

✅ 100-rate-metrics-100-times

Metric Status Value SLO % Under SLO
execution_time 1.700ms < 2.400ms +29.1%
max_rss_usage 29.137MB < 31.000MB +6.0%

✅ flush-1-metric

Metric Status Value SLO % Under SLO
execution_time 3.296µs < 10.000µs +67.0%
max_rss_usage 29.157MB < 31.000MB +5.9%

✅ flush-100-metrics

Metric Status Value SLO % Under SLO
execution_time 178.296µs < 200.000µs +10.9%
max_rss_usage 29.118MB < 31.000MB +6.1%

✅ flush-1000-metrics

Metric Status Value SLO % Under SLO
execution_time 2.204ms < 2.350ms +6.2%
max_rss_usage 29.884MB < 31.000MB +3.6%
tracer - 6/6

✅ large

Metric Status Value SLO % Under SLO
execution_time 29.686ms < 32.950ms +9.9%
max_rss_usage 30.297MB < 31.000MB +2.3%

✅ medium

Metric Status Value SLO % Under SLO
execution_time 2.896ms < 3.200ms +9.5%
max_rss_usage 28.705MB < 31.000MB +7.4%

✅ small

Metric Status Value SLO % Under SLO
execution_time 332.001µs < 370.000µs +10.3%
max_rss_usage 29.098MB < 31.000MB +6.1%

@P403n1x87 P403n1x87 force-pushed the chore/native-lock branch from 95a6cb7 to 214a5b3 Compare July 17, 2025 14:51
@DataDog DataDog deleted a comment from pr-commenter bot Jul 17, 2025
@P403n1x87 P403n1x87 force-pushed the chore/native-lock branch from 214a5b3 to 378288f Compare July 17, 2025 15:52
@P403n1x87 P403n1x87 marked this pull request as ready for review July 18, 2025 12:20
@P403n1x87 P403n1x87 requested review from a team as code owners July 18, 2025 12:20
@P403n1x87 P403n1x87 enabled auto-merge (squash) July 18, 2025 12:20
@P403n1x87 P403n1x87 requested a review from tylfin July 18, 2025 12:21
@P403n1x87 P403n1x87 changed the title chore: implement native lock objects refactor: implement native lock objects Jul 18, 2025
static inline void
Lock_reset(Lock* self)
{
self->_mutex = std::make_unique<std::timed_mutex>();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will call the destructor for the previous instance, which is UB if the mutex is locked (reference). I think we'd need to ensure that the mutex is unlocked for this to be safe, e.g. by locking in the parent, unlocking in the child. At which point, would resetting gain us anything?

If the goal is to just start clean regardless of the state of the mutex, I think you'd need to completely leak the old mutex and not call any methods on it. Like maybe by release-ing the old mutex from the unique_ptr and leaking it, and then creating a new one. Though if we just reset the mutex that may have been locked, that kinda implies that the parent process could have been in the middle of manipulating data protected by the mutex. Which could imply that the data isn't consistent in the child process...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will call the destructor for the previous instance, which is UB if the mutex is locked

Ah that's a good point!

locking in the parent, unlocking in the child

That's probably the best (and only real) alternative. The only issue is that if a lock is protecting a slow operation, the fork would be delayed. So it would be on us to ensure we don't lock on e.g. slow I/O.

I think you'd need to completely leak the old mutex and not call any methods on it

In practice this might be safe from a memory-leak perspective, as it is unlikely that one would want to fork from child processes, but it is nonetheless a possibility and a possible source of memory leaks 🤔 .

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trying to acquire the locks before a fork in the forking thread seems to lead to deadlocks in CI that I cannot reproduce locally. I think realistically leaking the lock is the only other viable option. This shouldn't be too different to the current implementation. For if we have a Python lock held by a thread (that is not the forking thread) by the time we fork, that thread will hold on to a reference of the lock forever. So even refreshing the lock object won't cause that lock to be GC'd

@P403n1x87 P403n1x87 force-pushed the chore/native-lock branch from cc2932d to 65c4899 Compare July 18, 2025 15:50
Copy link
Contributor

@taegyunkim taegyunkim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to take some of tests from https://github.com/taegyunkim/cpython/blob/main/Lib/test/lock_tests.py to test these locks?

@P403n1x87 P403n1x87 force-pushed the chore/native-lock branch 2 times, most recently from f9006ae to 75a7b4f Compare July 21, 2025 09:47
@tylfin tylfin requested a review from Copilot July 21, 2025 14:17
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements native lock objects to replace standard library-based locks throughout the ddtrace library. The change aims to isolate the library from potential patching of stdlib modules (e.g., by gevent) while significantly improving performance - from 1.05s to 0.14s in lock acquisition/release benchmarks.

Key changes include:

  • Introduction of native C++ Lock and RLock implementations in ddtrace.internal._threads
  • Systematic replacement of threading.Lock/RLock and forksafe.Lock/RLock with the new native implementations
  • Consolidation of thread management logic into a single ddtrace.internal.threads module

Reviewed Changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
ddtrace/internal/_threads.cpp Adds native C++ lock implementations and module registration
ddtrace/internal/_threads/lock.hpp Implements Lock and RLock classes with C++ std::mutex backing
ddtrace/internal/threads.py Creates unified thread module interface and fork safety registration
ddtrace/internal/forksafe.py Removes Lock/RLock factory functions (moved to threads module)
Multiple application files Updates imports from threading/forksafe locks to new threads module

Copy link
Member

@brettlangdon brettlangdon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not the right person to review lock.hpp, but everything else lgtm

@P403n1x87 P403n1x87 force-pushed the chore/native-lock branch from 75a7b4f to 34eb225 Compare July 21, 2025 15:05
@DataDog DataDog deleted a comment from Copilot AI Jul 21, 2025
@DataDog DataDog deleted a comment from Copilot AI Jul 21, 2025
@P403n1x87 P403n1x87 force-pushed the chore/native-lock branch 2 times, most recently from 68777f5 to 1f7e2d1 Compare July 22, 2025 11:18
@P403n1x87 P403n1x87 force-pushed the chore/native-lock branch from 1f7e2d1 to 6489d4a Compare July 22, 2025 11:56
@P403n1x87 P403n1x87 force-pushed the chore/native-lock branch from 6489d4a to 5ae4a34 Compare July 22, 2025 16:09
@P403n1x87 P403n1x87 force-pushed the chore/native-lock branch from c46d088 to e3e6599 Compare July 23, 2025 08:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
changelog/no-changelog A changelog entry is not required for this PR.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants