Skip to content

Add object name filters for SQL Server index usage and table size metrics - #24951

Merged
jasonmp85 merged 8 commits into
masterfrom
jmp/sqlserver-2a-object-bounds
Aug 25, 2026
Merged

Add object name filters for SQL Server index usage and table size metrics#24951
jasonmp85 merged 8 commits into
masterfrom
jmp/sqlserver-2a-object-bounds

Conversation

@jasonmp85

@jasonmp85 jasonmp85 commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds two optional configuration options that bound the table sets scanned by the two highest-fan-out database metric collectors:

  • index_usage_table_names, applied to index_usage_metrics
  • table_size_table_names, applied to table_size_metrics

When unset, behavior is unchanged. When set, the collector filters to the named tables. For index usage, these are table names—not index names—and metrics are collected for every index belonging to each matching table. This mirrors the existing db_fragmentation_object_names option, which provides the same escape hatch for the third high-fan-out collector; that legacy option also accepts table names despite its broader name. Postgres offers the equivalent lever as relations, whose own documentation warns that "patterns that match many relations can emit metrics in the thousands."

Measured effect

Sample volume scales with schema size, which is the fan-out being bounded (15-database pool):

tables/db tables indexes samples samples/db
2 45 120 809 53.9
20 315 930 5,129 341.9
100 1,515 4,530 24,329 1,621.9
300 4,515 13,530 72,329 4,821.9

Bounding on the 300-table pool (15 databases x 301 tables):

bound total samples table.row_count fragment_count
unset 72,328 4,515 13,530
10 names 2,488 150 435
3 names 808 45 120

And on a partitioned pool (5 databases x 10 tables, 1,200 index partitions), confirming the filter reaches partitioned indexes:

bound total samples table.row_count fragment_count
unset 5,078 50 1,200
3 names 1,578 15 360

Sample counts land exactly where the configuration predicts — 45 = 15 databases x 3 tables, 15 = 5 x 3, and fragmentation falls to 3/10 of unbounded on the partitioned pool. A 3-name bound cuts the wide pool by 89x.

Motivation

These collectors emit four samples per index and four per table respectively, giving a floor of roughly 28 samples per table before partitioning is taken into account. On instances with large schemas this produces both very large sample volumes and very large result sets, and row volume is the dominant cost: a separate experiment showed the check's dm_os_performance_counters query inflating 18.79x across 16 threads but only 2.25x across 16 processes with the server held constant, which places the serialization in CPython's GIL while pyodbc materializes rows rather than in SQL Server.

The failure this addresses is also directly reproducible. On a bench instance with 3,586 autodiscovered databases, an unmodified check produces:

Error querying sys.dm_db_index_usage_stats: [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]
There is insufficient system memory in resource pool 'default' to run this query. (701)

Bounding the table set is the only lever that reduces row volume without disabling the collector outright. Today a user can reach for that lever on fragmentation metrics but not on the other two, which is the inconsistency this closes.

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
  • If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

@datadog-prod-us1-6

datadog-prod-us1-6 Bot commented Aug 21, 2026

Copy link
Copy Markdown

Tests

All CI checks and tests passed.

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: b65c6e2 | Docs | View more details | Give us feedback!

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Aug 21, 2026

Copy link
Copy Markdown

evalya-impact-summary

evalya impact analysis
Impact analysis: RUN-ALL — every test task will run
Trigger:         empty diff (default branch, scheduled run, or shallow-clone fallback)
Test tasks:      0 (all selected)
Publish tasks:   2 (always emitted)
Diff:            empty (no diff information)

Learn more about CI impact filtering

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ef82d3c9d8

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread sqlserver/datadog_checks/sqlserver/database_metrics/index_usage_metrics.py Outdated
@jasonmp85
jasonmp85 force-pushed the jmp/sqlserver-2a-object-bounds branch 3 times, most recently from a98689d to 9e62ba0 Compare August 25, 2026 15:47
@jasonmp85 jasonmp85 added qa/skip-qa Automatically skip this PR for the next QA and removed qa/required QA is required for this PR and will generate a QA card labels Aug 25, 2026
@jasonmp85
jasonmp85 force-pushed the jmp/sqlserver-2a-object-bounds branch from 5962c11 to 60d008b Compare August 25, 2026 19:16
@jasonmp85
jasonmp85 force-pushed the jmp/sqlserver-2a-object-bounds branch from 267cd8b to b65c6e2 Compare August 25, 2026 19:31
@dd-octo-sts

dd-octo-sts Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Validation Report

All 21 validations passed.

Show details
Validation Description Status
agent-reqs Verify check versions match the Agent requirements file
ci Validate CI configuration and code coverage settings
codeowners Validate every integration has a CODEOWNERS entry
config Validate default configuration files against spec.yaml
dep Verify dependency pins are consistent and Agent-compatible
http Validate integrations use the HTTP wrapper correctly
imports Validate check imports do not use deprecated modules
integration-style Validate check code style conventions
jmx-metrics Validate JMX metrics definition files and config
labeler Validate PR labeler config matches integration directories
legacy-signature Validate no integration uses the legacy Agent check signature
license-headers Validate Python files have proper license headers
licenses Validate third-party license attribution list
metadata Validate metadata.csv metric definitions
models Validate configuration data models match spec.yaml
openmetrics Validate OpenMetrics integrations disable the metric limit
package Validate Python package metadata and naming
qa-label Validate the pull request declares whether it needs QA for the next Agent release
readmes Validate README files have required sections
saved-views Validate saved view JSON file structure and fields
version Validate version consistency between package and changelog

View full run

@jasonmp85
jasonmp85 enabled auto-merge August 25, 2026 19:42
@jasonmp85
jasonmp85 added this pull request to the merge queue Aug 25, 2026
Merged via the queue into master with commit 38ce99f Aug 25, 2026
104 checks passed
@jasonmp85
jasonmp85 deleted the jmp/sqlserver-2a-object-bounds branch August 25, 2026 19:52
@dd-octo-sts dd-octo-sts Bot added this to the 7.83.0 milestone Aug 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants