Skip to content

postgres: stream Remote Queries COPY output to the Agent relay - #24925

Merged
nubtron merged 2 commits into
nubtron/remote-queries-pocfrom
nubtron/remote-queries-intake-poc
Aug 27, 2026
Merged

postgres: stream Remote Queries COPY output to the Agent relay#24925
nubtron merged 2 commits into
nubtron/remote-queries-pocfrom
nubtron/remote-queries-intake-poc

Conversation

@nubtron

@nubtron nubtron commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Stack

  • Base: nubtron/remote-queries-poc
  • This draft: production Postgres COPY-to-Agent relay bridge

Summary

  • stream CSV COPY chunks synchronously through the native Agent callback
  • enforce bounded backpressure and result-delivery caps
  • return receipt-only metadata in upload mode
  • keep Agent-owned credentials and intake routing out of Python
  • preserve the ordinary small-result behavior when upload mode is omitted

Validation

  • 129 focused tests passed
  • lint passed
  • independently reviewed

Deployment

Used only by the local Remote Queries intake proof setup; no shared deployment.

Extend the Postgres remote query COPY stream with an optional
resultDelivery instruction (mode POC_PUBLIC_CHUNKED_UPLOAD). When
present, the integration feeds bounded COPY bytes through the
existing native emit callback to the Agent-owned upload transport:
chunks are sized by resultDelivery.chunkBytes, capped by
resultDelivery.maxBytes, and carry a SHA-256 digest; the STARTED
event carries the sanitized upload-session handle (uploadId only) and
the final event carries a compact provisional uploadReceipt aligned
to the Agent-owned receipt shape. Python never receives the upload
baseUrl, scoped token, or API/app keys and performs no HTTP upload;
the Agent Go side retains those secrets. Omitting resultDelivery
keeps the inline streaming behavior unchanged.
@nubtron nubtron added the qa/skip-qa Automatically skip this PR for the next QA label Aug 20, 2026
@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Aug 20, 2026

Copy link
Copy Markdown

Tests  Code Coverage

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 85.30%
Overall Coverage: 93.95% (-0.28%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: bf9d874 | Docs | View more details | Give us feedback!

…tion

Replace the emit-callback upload bridge with direct HTTP upload to
its-agent-intake from the Postgres Python integration. When resultDelivery is
present, the integration PUTs bounded COPY chunks straight to the intake over
HTTP using the base URL and scoped upload token forwarded through the native
bridge plus the org API key and POC application key read from Agent config
via datadog_agent.get_config.

Bulk chunk bytes no longer traverse the native emit bridge, AgentSecure, PAR,
or AP action output; only metadata/final/error events cross back. The
integration owns chunking, per-chunk sha256, the HTTP PUT with retry on
transient 408/429/5xx, finalize, and best-effort abort on failure.

RemoteQueryResultDelivery now accepts and requires baseUrl and token.
execute_agent_rpc_stream_copy routes upload-mode requests to a new
_execute_upload_stream that drives the existing chunk iterator but sinks data
events to an injectable _UploadClient (production uses requests; tests inject
a fake). Omitting resultDelivery keeps the inline streaming path unchanged.
@dd-octo-sts

dd-octo-sts Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Validation Report

All 21 validations passed.

Show details
Validation Description Status
agent-reqs Verify check versions match the Agent requirements file
ci Validate CI configuration and code coverage settings
codeowners Validate every integration has a CODEOWNERS entry
config Validate default configuration files against spec.yaml
dep Verify dependency pins are consistent and Agent-compatible
http Validate integrations use the HTTP wrapper correctly
imports Validate check imports do not use deprecated modules
integration-style Validate check code style conventions
jmx-metrics Validate JMX metrics definition files and config
labeler Validate PR labeler config matches integration directories
legacy-signature Validate no integration uses the legacy Agent check signature
license-headers Validate Python files have proper license headers
licenses Validate third-party license attribution list
metadata Validate metadata.csv metric definitions
models Validate configuration data models match spec.yaml
openmetrics Validate OpenMetrics integrations disable the metric limit
package Validate Python package metadata and naming
qa-label Validate the pull request declares whether it needs QA for the next Agent release
readmes Validate README files have required sections
saved-views Validate saved view JSON file structure and fields
version Validate version consistency between package and changelog

View full run

@cit-pr-commenter-54b7da

Copy link
Copy Markdown

evalya-impact-summary

evalya impact analysis
Impact analysis: 0 selected, 0 skipped (of 0 test tasks)
Publish tasks:   1 (always emitted)
Diff (6 files):
  postgres/changelog.d/23499.added
  postgres/changelog.d/24311.added
  postgres/changelog.d/24311.fixed
  postgres/datadog_checks/postgres/remote_query.py
  postgres/tests/test_remote_query.py
  postgres/tests/test_remote_query_integration.py

Debug a specific task: evalya plan impact --path <path> --task <task>

Learn more about CI impact filtering

@nubtron
nubtron merged commit bf9d874 into nubtron/remote-queries-poc Aug 27, 2026
40 of 60 checks passed
@nubtron
nubtron deleted the nubtron/remote-queries-intake-poc branch August 27, 2026 15:11
@nubtron
nubtron restored the nubtron/remote-queries-intake-poc branch August 27, 2026 15:13

nubtron commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator Author

Consolidated into #23499 (draft). The stack is linear, so #23499 fast-forwards to the hosted #24926 tree, preserving these commits. GitHub auto-merged this draft when the #23499 base advanced through the stack; the head branch has been restored and remains available for later explicit cleanup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

integration/postgres qa/skip-qa Automatically skip this PR for the next QA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant