fix(aws_S3 source): Add timeout for aws operations. by petere-datadog · Pull Request #25873 · vectordotdev/vector

petere-datadog · 2026-07-16T16:31:37Z

Summary

Update all amazon integrations to include some default timeouts

Change Type

Is this a breaking change?

Yes
No

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the no-changelog label to this PR.

References

Notes

Please read our Vector contributor resources.
Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
Some CI checks run only after we manually approve them.
- We recommend adding a pre-push hook, please see this template.
- Alternatively, we recommend running the following locally before pushing to the remote branch:
  - make fmt
  - make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
  - make test
After a review is requested, please avoid force pushes to help us review incrementally.
- Feel free to push as many commits as you want. They will be squashed into one before merging.
- For example, you can run git merge origin master and git push.
If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
run make build-licenses to regenerate the license inventory and commit the changes (if any). More details on the dd-rust-license-tool.

The aws_s3 source created its S3 client with no read or operation timeout (only the SDK-default ~3.1s connect timeout). A half-open or silently dropped connection -- for example one reaped by a NAT gateway idle timeout -- could therefore hang a polling task indefinitely on the GetObject response-body read, stopping SQS polling entirely and stalling S3-based ingestion until the process was restarted. Retries are disabled on this client, so there was no recovery. Apply default S3 client timeouts (connect_timeout_seconds=5, read_timeout_seconds=30) and expose them as source configuration. The read timeout is applied per-read, so a slow but steadily progressing transfer of a large object is not cut off; no operation timeout is set by default to avoid capping legitimately long transfers. This mirrors the existing SQS client timeout support on the same source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ik/aws-read-timeout

Error logs for failed S3 events only included the SQS message ID, making it hard to tell which S3 object actually failed without cross-referencing. Log the object key (when known) alongside the error at error level.

datadog-vectordotdev · 2026-07-16T16:32:14Z

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: f531ef2 | Docs | Give us feedback!}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 644e4cd109

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 273a0fd98a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

graphcareful · 2026-07-20T15:52:09Z

+    AwsTimeout::new(
+        configured
+            .connect_timeout()
+            .or(Some(DEFAULT_S3_CONNECT_TIMEOUT_SECONDS)),


Question is this default value how the code behaved previously? If not then maybe we would consider doing nothing if the optional is not filled. Many other integrations do this instead of falling back to a hardcoded default.

So the default value before was 3.1 seconds https://github.com/awslabs/aws-sdk-rust/blob/33a43d1891b5ae5f5c3ed83343193967c7e01446/sdk/aws-smithy-runtime/src/client/defaults.rs#L38 now we're setting it to 5 seconds. Read timeout wasn't set at all so we're setting it to 30seconds which I think is a good default, cause if the connection is stuck, i.e we've connected but no data is coming across then we shouldn't hang indefinitely.

Hmm i see. This can go down two paths either:

we are ok with whatever default the provider ships in their library

we are not ok and we think 5s is a better choice.

If we go with (1) then my previous suggestion works. If (2) then I think the conventional way to do this in vector is with the #[serde(default) = "xyz"] configurable component tag.

alright made a larger update, looks like a lot but it's all kinda samey, decided to add timeouts to all the other AWS integrations as well.

…ents Extend the connect/operation/read timeout options previously only available on the aws_s3 source to the aws_cloudwatch_logs, aws_cloudwatch_metrics, aws_kinesis_firehose, aws_kinesis_streams, aws_s3, aws_sns, and aws_sqs sinks, the aws_sqs source, and the aws_secrets_manager secrets backend (as client_timeout, to avoid a naming collision with the exec backend's unrelated timeout field). Also fixes AwsTimeout's Default impl to match its serde field defaults (connect_timeout_seconds=5, read_timeout_seconds=30) instead of silently defaulting to 0.

…ik/aws-read-timeout

Fold the aws_s3-source-only fix fragment into the enhancement fragment now that timeout settings cover all AWS-backed components.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4ef846b98d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-07-20T19:57:11Z

+    timeout_config_builder
+        .set_operation_timeout(operation_timeout.map(Duration::from_secs))
+        .set_connect_timeout(Some(Duration::from_secs(connect_timeout)))
+        .set_read_timeout(Some(Duration::from_secs(read_timeout)));


Avoid forcing first-byte timeout on slow uploads

With the new defaults, every AWS client now gets a 30s read_timeout even when the component config does not opt in. That timeout is documented in AwsTimeout as time to the first response byte from request start, so for sinks that upload request bodies—especially aws_s3 with large batches or slow/S3-compatible endpoints—the response headers may legitimately arrive only after more than 30s of uploading/service work, causing previously valid writes to time out and retry indefinitely unless users discover and raise this new setting. Consider not applying a default read timeout to upload-oriented clients, or making the default opt-in/large enough for existing sink workloads.

Useful? React with 👍 / 👎.

taylorchandleryoung and others added 3 commits July 16, 2026 15:55

Merge branch 'master' of github.com:vectordotdev/vector into peter.eh…

efc4dd3

…ik/aws-read-timeout

fix(aws_s3 source): log S3 object key on processing errors

644e4cd

Error logs for failed S3 events only included the SQS message ID, making it hard to tell which S3 object actually failed without cross-referencing. Log the object key (when known) alongside the error at error level.

petere-datadog requested a review from a team as a code owner July 16, 2026 16:31

github-actions Bot added the domain: sources Anything related to the Vector's sources label Jul 16, 2026

petere-datadog changed the title ~~fix(aws_S3): Add timeout for s3 operations. Add more error logging when get object fails.~~ fix(aws_S3 source): Add timeout for s3 operations. Add more error logging when get object fails. Jul 16, 2026

chatgpt-codex-connector Bot reviewed Jul 16, 2026

View reviewed changes

Comment thread src/sources/aws_s3/mod.rs Outdated

petere-datadog added 2 commits July 16, 2026 13:04

remove this enhancement.md file

9fcfa4d

add timeout settings

273a0fd

petere-datadog requested a review from a team as a code owner July 16, 2026 18:22

github-actions Bot added docs review on hold The documentation team reviews PRs only after a PR is approved by the COSE team. domain: external docs Anything related to Vector's external, public documentation labels Jul 16, 2026

chatgpt-codex-connector Bot reviewed Jul 16, 2026

View reviewed changes

Comment thread src/sources/aws_s3/mod.rs Outdated

petere-datadog added 2 commits July 20, 2026 11:46

remove uneeded logging

55ace27

no-op

ff1d80e

graphcareful reviewed Jul 20, 2026

View reviewed changes

petere-datadog added 2 commits July 20, 2026 15:44

Merge branch 'master' of github.com:vectordotdev/vector into peter.eh…

f2814ae

…ik/aws-read-timeout

github-actions Bot added the domain: sinks Anything related to the Vector's sinks label Jul 20, 2026

petere-datadog changed the title ~~fix(aws_S3 source): Add timeout for s3 operations. Add more error logging when get object fails.~~ fix(aws_S3 source): Add timeout for aws operations. Jul 20, 2026

petere-datadog added 2 commits July 20, 2026 15:47

chore(changelog): consolidate AWS timeout changelog fragments

6b19b96

Fold the aws_s3-source-only fix fragment into the enhancement fragment now that timeout settings cover all AWS-backed components.

update changelog description

4ef846b

chatgpt-codex-connector Bot reviewed Jul 20, 2026

View reviewed changes

fixed changelog

f531ef2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(aws_S3 source): Add timeout for aws operations.#25873

fix(aws_S3 source): Add timeout for aws operations.#25873
petere-datadog wants to merge 12 commits into
masterfrom
peter.ehik/aws-read-timeout

petere-datadog commented Jul 16, 2026 •

edited

Loading

Uh oh!

datadog-vectordotdev Bot commented Jul 16, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

graphcareful Jul 20, 2026

Uh oh!

petere-datadog Jul 20, 2026

Uh oh!

graphcareful Jul 20, 2026

Uh oh!

petere-datadog Jul 20, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

petere-datadog commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Change Type

Is this a breaking change?

Does this PR include user facing changes?

References

Notes

Uh oh!

datadog-vectordotdev Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

graphcareful Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

petere-datadog Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

graphcareful Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

petere-datadog Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

petere-datadog commented Jul 16, 2026 •

edited

Loading

datadog-vectordotdev Bot commented Jul 16, 2026 •

edited

Loading