Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve handling of streams using context manager #419

Open
jackmpcollins opened this issue Feb 2, 2025 · 0 comments
Open

Improve handling of streams using context manager #419

jackmpcollins opened this issue Feb 2, 2025 · 0 comments

Comments

@jackmpcollins
Copy link
Owner

OpenAI and Anthropic packages both use context managers to ensure the stream for the streamed response is closed correctly. magentic should do the same!

Add a new Stream (and AsyncStream) class, that is an Iterable with an additional .close() method and ability to be used as a context manager (returning self and automatically calling .close() on exit). This should be usable as a return type on prompt-functions.

StreamedStr and StreamedResponse should also work similarly.

Maybe when the Iterable return type is used, we should return a Stream instance.

Example usage:

from magentic import prompt, StreamedStr


@prompt("Tell me about {country}")
def describe_country(country: str) -> StreamedStr: ...


# Print the chunks while they are being received
with describe_country("Brazil") as stream:
    for chunk in stream:
        print(chunk, end="")
# 'Brazil, officially known as the Federative Republic of Brazil, is ...'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant