Batch SQL Server per-database queries

[jasonmp85]

What does this PR do?

Collapses two SQL Server operations that previously scaled with the number of autodiscovered databases:

• Database connection service checks now use one query over the monitored database names and derive the same per-database statuses from its result.
• Database file metrics now use one compact TDS request. A server-side cursor invokes the original sys.database_files query through [database].sys.sp_executesql, preserving each database's execution context, and the client drains the streamed result sets with nextset().

The database file query still uses FILEPROPERTY(name, 'SpaceUsed') inside each database, so it preserves current tempdb sizes and does not require VIEW ANY DEFINITION. Database names are supplied as an XML parameter and quoted with QUOTENAME. A database that becomes unavailable after autodiscovery is skipped so later databases are still collected.

Azure SQL Database retains the direct database-scoped query because it does not support cross-database execution. Azure SQL Managed Instance uses the batched path.

Motivation

The previous implementation issued two SQL requests per autodiscovered database for each affected path. Both paths were benchmarked through the real SSH tunnel against the same SQL Server 2022 instance with 3,945 databases, with one workload running at a time:

collection	previous requests	final requests	previous wall time	final wall time	speedup
Database connection status	7,891	1	327.936 s	0.353 s median	929.1x
Database file metrics	7,890	1	368.700 s	52.000 s	7.1x
Combined scoped work	15,781	2	696.636 s	52.353 s	13.31x

This removes 15,779 requests (99.987%) and saves 644.283 seconds, or 10 minutes 44.3 seconds, per collection at this scale. The combined result is the arithmetic sum of separately measured components under the same tunnel and database population, not a separate end-to-end run.

The status benchmark ran the final path four times around one full legacy sweep. Final timings ranged from 0.334 to 0.425 seconds; the median was 0.353 seconds and the first row arrived in 0.134 to 0.186 seconds. A server-local control measured 2.086 seconds for the legacy path and 0.243 to 0.288 seconds for the final path, confirming that the large tunneled improvement comes primarily from eliminating round trips.

Correctness was compared against the original behavior across all 3,945 databases: every connection status matched, and database file coverage and every stable value matched across 7,897 rows. Restricted-permission probes also matched for accessible, inaccessible, and nonexistent databases. Database file collection was verified with CONNECT ANY DATABASE but not VIEW ANY DEFINITION, including rows from regular, partitioned, and tempdb databases. One-page changes observed in log space_used values followed whichever query ran later and were confirmed as live checkpoint activity rather than a code-path difference.

Focused tests cover bounded query count, multi-result draining, and the Azure SQL Database fallback. Integration tests compare database file size, used space, and state with database-local sys.database_files results and verify collection under restricted permissions. The full Linux ODBC/FreeTDS and Windows ODBC/OLE DB CI matrix passes.

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
• If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

[datadog-datadog-prod-us1]

 

⚠️ Warnings

❄️ 1 New flaky test detected

test_database_files_metrics_match_database_context from test_integration.py

assert {('master', 1): 4608.0, ('master', 2): 2048.0, ('msdb', 1): 15168.0, ('msdb', 2): 1024.0, ('tempdb', 1): 8192.0, ('tempdb', 2): 8192.0, ('tempdb', 3): 8192.0, ('tempdb', 4): 8192.0, ('tempdb', 5): 8192.0} == {('master', 1): 4608, ('master', 2): 2048, ('msdb', 1): 15168, ('msdb', 2): 768, ('tempdb', 1): 8192, ('tempdb', 2): 8192, ('tempdb', 3): 8192, ('tempdb', 4): 8192, ('tempdb', 5): 8192}
  
  Common items:
  {('master', 1): 4608.0,
   ('master', 2): 2048.0,
   ('msdb', 1): 15168.0,
   ('tempdb', 1): 8192.0,
   ('tempdb', 2): 8192.0,
   ('tempdb', 3): 8192.0,
   ('tempdb', 4): 8192.0,
...

View in Flaky Test Management

ℹ️ Info

No other issues found (see more)

🧪 All tests passed

🎯 Code Coverage (details)
• Patch Coverage: 97.32%
• Overall Coverage: 90.80% (+2.13%)

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 7cad165 | Docs | View more details | Give us feedback!}

[jasonmp85]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f900b05f1f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[cit-pr-commenter-54b7da]

evalya-impact-summary

evalya impact analysis

Impact analysis: 0 selected, 0 skipped (of 0 test tasks)
Publish tasks:   1 (always emitted)
Diff (8 files):
  sqlserver/changelog.d/24950.fixed
  sqlserver/datadog_checks/sqlserver/const.py
  sqlserver/datadog_checks/sqlserver/database_metrics/database_files_metrics.py
  sqlserver/datadog_checks/sqlserver/sqlserver.py
  sqlserver/datadog_checks/sqlserver/utils.py
  sqlserver/tests/test_database_metrics.py
  sqlserver/tests/test_integration.py
  sqlserver/tests/test_unit.py

Debug a specific task: evalya plan impact --path <path> --task <task>

Learn more about CI impact filtering

[jasonmp85]

I ran a temporary end-to-end test against a real Azure SQL Database. It asserted that Azure returned EngineEdition = 5, that the integration selected the database-scoped executor with db=pr24950, and that the emitted file size, used-space, state, db, and database values matched a direct sys.database_files query.

tests/test_live_azure_pr24950.py::test_live_azure_database_files PASSED [100%]
1 passed in 7.77s

Review from sethsamuel is dismissed. Related teams and files:

• database-monitoring-agent
  • sqlserver/tests/test_integration.py

[dd-octo-sts]

Validation Report

All 21 validations passed.

Show details

Validation	Description	Status
agent-reqs	Verify check versions match the Agent requirements file	✅
ci	Validate CI configuration and code coverage settings	✅
codeowners	Validate every integration has a CODEOWNERS entry	✅
config	Validate default configuration files against spec.yaml	✅
dep	Verify dependency pins are consistent and Agent-compatible	✅
http	Validate integrations use the HTTP wrapper correctly	✅
imports	Validate check imports do not use deprecated modules	✅
integration-style	Validate check code style conventions	✅
jmx-metrics	Validate JMX metrics definition files and config	✅
labeler	Validate PR labeler config matches integration directories	✅
legacy-signature	Validate no integration uses the legacy Agent check signature	✅
license-headers	Validate Python files have proper license headers	✅
licenses	Validate third-party license attribution list	✅
metadata	Validate metadata.csv metric definitions	✅
models	Validate configuration data models match spec.yaml	✅
openmetrics	Validate OpenMetrics integrations disable the metric limit	✅
package	Validate Python package metadata and naming	✅
qa-label	Validate the pull request declares whether it needs QA for the next Agent release	✅
readmes	Validate README files have required sections	✅
saved-views	Validate saved view JSON file structure and fields	✅
version	Validate version consistency between package and changelog	✅

View full run

[jasonmp85]

I fixed a flaky test and it dismissed your review :-|