fix(streaming): ensure remaining body is consumed after [DONE] in Str… - #3581
fix(streaming): ensure remaining body is consumed after [DONE] in Str…#3581vrs-darkness wants to merge 2 commits into
Conversation
…eam and AsyncStream - Added best-effort draining of remaining body bytes after receiving [DONE] to allow connection reuse. - Implemented error handling to prevent stream failures due to transport errors during draining. - Introduced tests to validate the behavior for both synchronous and asynchronous streams.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d4b47e7bb6
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…and AsyncStream - Updated error handling in both Stream and AsyncStream classes to catch UnicodeError in addition to HTTPError during the draining of the stream after receiving [DONE]. - Added a new test to ensure that malformed trailing bytes after [DONE] do not cause failures in already-complete streams for both synchronous and asynchronous scenarios.
|
@codex review |
|
Codex Review: Didn't find any major issues. Swish! Reviewed commit: ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
|
Security review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
|
@jbeckwith-oai please review PR whenever possible. It is currently a patch in my service would be great if you could check the PR and take it for a release which i could use directly. |
1 participant
fix(streaming): best-effort drain after [DONE] for connection reuse
Changes being requested
When using a connection-pooled OpenAI client, each streaming request was adding ~300ms of overhead. SSE streams break on
data: [DONE]without fully reading the remaining response body; closing that partially-read response drops the connection from the pool, so the next request pays for a new TCP/TLS handshake.After
[DONE], drain the remaining iterator withconsume_sync_iterator/consume_async_iteratorbefore close so the body can be fully consumed and the pooled connection reused.Drain is best-effort:
httpx.HTTPErrorduring cleanup is swallowed so[DONE]stays terminal for callers even if a proxy leaves trailing heartbeats, the socket errors, or trailing bytes are truncated/malformed.