perf: stream request body #103
Conversation
Signed-off-by: Danny Kopping <[email protected]>
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite. This stack of pull requests is managed by Graphite. Learn more about stacking. |
| case routeMessages: | ||
| var req messages.MessageNewParamsWrapper | ||
| if err := json.Unmarshal(payload, &req); err != nil { | ||
| if err := json.NewDecoder(r.Body).Decode(&req); err != nil { |
There was a problem hiding this comment.
I tried this when I ran the initial scaletests in my workspace.
It streams top level, which is great. But the moment it descends into the Anthropic/OpenAI stainless sdk, it just Unmarshals there anyway. I wasn't able to measure any tangible improvements by doing this. Is there something that I'm missing?
There was a problem hiding this comment.
I'd love a world where we can stream the entire payload a single token at a time and then essentially tee that stream into the database for analytics, but I haven't found a way to do that while we use the SDKs that we're currently using.
2 participants
TODO: apply to OpenAI as well.