Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: remove ability to make tags on a context #437

Merged
merged 1 commit into from
Jan 27, 2025

Conversation

tobz
Copy link
Member

@tobz tobz commented Jan 23, 2025

Context

We don't currently do anything with mutating the tags on a context, at least not anymore... so removing this for now to clean things up.

@pr-commenter
Copy link

pr-commenter bot commented Jan 23, 2025

Regression Detector (DogStatsD)

Regression Detector Results

Run ID: 091388f7-e2c9-44a7-a96a-864708896a5c

Baseline: 7.62.0-rc.8
Comparison: 7.62.0-rc.8

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment

perf experiment goal Δ mean % Δ mean % CI trials links
dsd_uds_10mb_3k_contexts ingress throughput +0.00 [-0.00, +0.01] 1
dsd_uds_100mb_3k_contexts ingress throughput +0.00 [-0.05, +0.05] 1
dsd_uds_512kb_3k_contexts ingress throughput +0.00 [-0.01, +0.01] 1
dsd_uds_1mb_3k_contexts ingress throughput +0.00 [-0.00, +0.00] 1
dsd_uds_100mb_250k_contexts ingress throughput +0.00 [-0.00, +0.00] 1
dsd_uds_1mb_50k_contexts_memlimit ingress throughput -0.00 [-0.00, +0.00] 1
dsd_uds_1mb_50k_contexts ingress throughput -0.00 [-0.00, +0.00] 1
dsd_uds_1mb_3k_contexts_dualship ingress throughput -0.00 [-0.00, +0.00] 1
dsd_uds_40mb_12k_contexts_40_senders ingress throughput -0.00 [-0.00, +0.00] 1
dsd_uds_500mb_3k_contexts ingress throughput -0.00 [-0.01, +0.01] 1
dsd_uds_100mb_3k_contexts_distributions_only memory utilization -1.26 [-1.47, -1.06] 1
quality_gates_idle_rss memory utilization -1.52 [-1.60, -1.44] 1

Bounds Checks: ❌ Failed

perf experiment bounds_check_name replicates_passed links
quality_gates_idle_rss memory_usage 0/10

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

  • ✅ = significantly better comparison variant performance
  • ❌ = significantly worse comparison variant performance
  • ➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

  1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

  2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

  3. Its configuration does not mark it "erratic".

@pr-commenter
Copy link

pr-commenter bot commented Jan 23, 2025

Regression Detector (Saluki)

Regression Detector Results

Run ID: 5f7835f3-77f8-472f-ad2d-e27cdeb0f49a

Baseline: 507fe24
Comparison: df12b22
Diff

❌ Experiments with missing or malformed data

This is a critical error. No usable optimization goal data was produced by the listed experiments. This may be a result of misconfiguration. Ping #single-machine-performance and we can help out.

  • dsd_uds_100mb_250k_contexts

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment

perf experiment goal Δ mean % Δ mean % CI trials links
dsd_uds_500mb_3k_contexts ingress throughput +0.83 [+0.71, +0.95] 1
dsd_uds_1mb_50k_contexts_memlimit ingress throughput +0.20 [-0.28, +0.67] 1
dsd_uds_10mb_3k_contexts ingress throughput +0.02 [-0.00, +0.05] 1
dsd_uds_50mb_10k_contexts_no_inlining ingress throughput +0.01 [-0.06, +0.08] 1
dsd_uds_50mb_10k_contexts_no_inlining_no_allocs ingress throughput +0.00 [-0.06, +0.06] 1
dsd_uds_1mb_50k_contexts ingress throughput +0.00 [-0.00, +0.01] 1
dsd_uds_512kb_3k_contexts ingress throughput -0.00 [-0.01, +0.01] 1
dsd_uds_1mb_3k_contexts ingress throughput -0.00 [-0.00, +0.00] 1
dsd_uds_100mb_3k_contexts ingress throughput -0.00 [-0.05, +0.04] 1
quality_gates_idle_rss memory utilization -0.01 [-0.04, +0.02] 1
dsd_uds_40mb_12k_contexts_40_senders ingress throughput -0.01 [-0.04, +0.02] 1
dsd_uds_1mb_3k_contexts_dualship ingress throughput -0.02 [-0.03, +0.00] 1
dsd_uds_100mb_3k_contexts_distributions_only memory utilization -0.97 [-1.10, -0.84] 1

Bounds Checks: ✅ Passed

perf experiment bounds_check_name replicates_passed links
quality_gates_idle_rss memory_usage 10/10

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

  • ✅ = significantly better comparison variant performance
  • ❌ = significantly worse comparison variant performance
  • ➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

  1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

  2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

  3. Its configuration does not mark it "erratic".

@pr-commenter
Copy link

pr-commenter bot commented Jan 23, 2025

Regression Detector Links

Experiment Result Links

experiment link(s)
dsd_uds_100mb_250k_contexts [Profiling (ADP)] [Profiling (DSD)] [SMP Dashboard]
dsd_uds_100mb_3k_contexts [Profiling (ADP)] [Profiling (DSD)] [SMP Dashboard]
dsd_uds_100mb_3k_contexts_distributions_only [Profiling (ADP)] [Profiling (DSD)] [SMP Dashboard]
dsd_uds_10mb_3k_contexts [Profiling (ADP)] [Profiling (DSD)] [SMP Dashboard]
dsd_uds_1mb_3k_contexts [Profiling (ADP)] [Profiling (DSD)] [SMP Dashboard]
dsd_uds_1mb_3k_contexts_dualship [Profiling (ADP)] [Profiling (DSD)] [SMP Dashboard]
dsd_uds_1mb_50k_contexts [Profiling (ADP)] [Profiling (DSD)] [SMP Dashboard]
dsd_uds_1mb_50k_contexts_memlimit [Profiling (ADP)] [Profiling (DSD)] [SMP Dashboard]
dsd_uds_40mb_12k_contexts_40_senders [Profiling (ADP)] [Profiling (DSD)] [SMP Dashboard]
dsd_uds_500mb_3k_contexts [Profiling (ADP)] [Profiling (DSD)] [SMP Dashboard]
dsd_uds_512kb_3k_contexts [Profiling (ADP)] [Profiling (DSD)] [SMP Dashboard]
quality_gates_idle_rss [Profiling (ADP)] [Profiling (DSD)] [SMP Dashboard]
dsd_uds_50mb_10k_contexts_no_inlining (ADP only) [Profiling (ADP)] [SMP Dashboard]
dsd_uds_50mb_10k_contexts_no_inlining_no_allocs (ADP only) [Profiling (ADP)] [SMP Dashboard]

@tobz tobz force-pushed the tobz/remove-context-tags-mutability branch from df12b22 to de9f7e9 Compare January 27, 2025 14:51
@tobz tobz merged commit 3576732 into main Jan 27, 2025
9 of 14 checks passed
@tobz tobz deleted the tobz/remove-context-tags-mutability branch January 27, 2025 14:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/core Core functionality, event model, etc.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants