postgres: add deterministic Remote Queries COPY relay proofs - #24926
Draft
nubtron wants to merge 5 commits into
Draft
postgres: add deterministic Remote Queries COPY relay proofs#24926nubtron wants to merge 5 commits into
nubtron wants to merge 5 commits into
Conversation
Contributor
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 864bad8 | Docs | View more details | Give us feedback! |
…ad proof The Postgres COPY bridge now uploads chunks directly to its-agent-intake over HTTP instead of feeding bytes through the native emit callback. Replace the obsolete emit-bridge proof (which asserted data events arrive at the native callback with sha256 and one-chunk-in-flight) with a direct-HTTP upload proof. The new proof drives the real _execute_upload_stream with an injectable FakeUploadClient and asserts: - Bulk bytes go directly to the intake over HTTP (fake put_chunk calls) in 1 MiB bounds, not through the emit callback; only metadata/final/error cross back. - One-chunk backpressure: COPY reads are fenced during each HTTP upload (lockstep reads_at_put == 1..N), so the full payload is never buffered ahead. - Per-chunk SHA-256 matches the payload and the aggregate matches the incremental reference at 8 MiB and 32 MiB. - Finalize is called once and the camelCase receipt surfaces under the snake_case upload_receipt key. - HTTP failure aborts the session and emits an error event; maxBytes is enforced at MiB scale. The 243021 task reference is preserved as the deterministic upload session id.
nubtron
force-pushed
the
nubtron/remote-queries-intake-tooling
branch
from
August 20, 2026 16:30
fd725cc to
657992e
Compare
evalya-impact-summaryevalya impact analysis |
… contract
Replace the Postgres Remote Queries chunk-object/manifest uploader with a
sequential multipart HTTP client using the POC_PUBLIC_MULTIPART_UPLOAD
contract. The integration aggregates COPY output into at most one configured
partBytes buffer and uploads contiguous 1-based /parts/{part_number} requests
carrying X-DD-Part-SHA256/Bytes/Rows headers, retries the same part
idempotently, finalizes once, and returns the small objectPath/partCount
receipt. A final short part is allowed and a zero-row result finalizes with
zero parts. Overall stream SHA-256, maxBytes enforcement, abort behavior,
credential redaction, and no bulk Agent output are preserved.
Delete the old POC_PUBLIC_CHUNKED_UPLOAD mode and the chunkBytes/manifestPath/
chunkCount aliases and compatibility. Add tests for the exact HTTP contract,
idempotent retry, fail-closed rejection, multipart sizing with a final short
part, and zero-row finalization.
…gregate chunks into parts partBytes (the multipart part size) and limits.chunkBytes (the COPY streaming chunk size) are distinct concepts. The worker injects an 8 MiB partBytes while the COPY chunk size defaults to 1 MiB, so the previous cross-field clamp (resultDelivery.partBytes <= limits.chunkBytes) rejected every normal multipart dispatch. Remove that clamp; keep resultDelivery.maxBytes <= limits.maxBytes. The COPY stream now always emits limits.chunkBytes-sized events. The upload client aggregates those events into partBytes-sized parts via a bounded _MultipartBuffer (at most one part buffered), computes each part's SHA-256 over the aggregated body, and uploads contiguous 1-based parts. The STARTED metadata reports the COPY chunkBytes and the resultDelivery partBytes separately; the provisional receipt reports the part count (ceil(totalBytes/partBytes)), not the chunk count. The no-compat multipart contract is preserved. Add regressions proving partBytes may exceed limits.chunkBytes and that the upload client aggregates COPY chunks into one partBytes part (with a final short part), plus that each part carries the SHA-256 of its aggregated body.
…d 5-minute HTTP timeout Raise the POC multipart upload capacity and timeouts to match the plan's larger result envelope. resultDelivery.maxBytes is now accepted up to a 10 GiB server-owned maximum (and still must not exceed limits.maxBytes), and partBytes up to 128 MiB, independent of the 1 MiB COPY read chunk. The COPY stream still emits limits.chunkBytes-sized events that the upload client aggregates into partBytes parts, so memory stays bounded to one multipart part. Change the per-upload HTTP timeout from a 60-second scalar to an explicit (connect, read) tuple with a 10-second connect timeout and a 5-minute read timeout so a slow large-part upload is not cut short while a stuck connect fails fast. The timeout is now plumbed through _RequestsUploadClient into _upload_with_retry (the constructor parameter was previously stored but unused); retry count and backoff stay bounded. Add regressions proving the 10 GiB maxBytes boundary is accepted when the extraction cap matches and rejected one byte past it, the 128 MiB partBytes boundary, that many 1 MiB COPY reads aggregate into 64 MiB parts without materializing the full result, and that the HTTP timeout is the 5-minute (connect, read) tuple. No test allocates 10 GiB. Same-part retry, final short part, maxBytes fail closed, abort, checksums, and no bulk Agent output are preserved.
The intake finalize route serializes the canonical final path as the snake-case object_path field, but _intake_receipt_to_camel read the obsolete object_key, producing an empty AP receipt objectPath on the multipart finalization path. Map object_path to objectPath instead. Remote Queries is greenfield, so the object_key alias is not preserved. Update the finalize fixtures to object_path and add focused unit tests covering the mapping and the dropped alias.
Contributor
Validation ReportAll 21 validations passed. Show details
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack
nubtron/remote-queries-intake-poc)Summary
remote_query.pyunchanged in this tooling layerValidation
Deployment
Test-only local proof tooling; no shared deployment.