Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support store and metadata fields for OpenAI evals/distillation #423

Open
jackmpcollins opened this issue Feb 2, 2025 · 0 comments
Open

Comments

@jackmpcollins
Copy link
Owner

Add support for OpenAI evals and distillation https://platform.openai.com/docs/guides/evals
https://platform.openai.com/docs/guides/distillation

Add ability to provide store and metadata parameters to OpenaiChatModel defaulting to NOT_GIVEN. If store is set to true, the prompt-function name should be added as a metadata item - this might mean metadata will need to be provided in OpenaiChatModel.complete.

Also to be decided: provide all the prompt-function arguments as metadata? (key = param name, value = argument). Would have to filter out those not meeting the limits of 512 chars. These might be useful to have judging by the video on the docs page showing data being inserted into a prompt template (just like magentic prompt-functions do). Need to try out the evals product to confirm this.

    metadata: Optional[Metadata]
    """Set of 16 key-value pairs that can be attached to an object.

    This can be useful for storing additional information about the object in a
    structured format, and querying for objects via API or the dashboard.

    Keys are strings with a maximum length of 64 characters. Values are strings with
    a maximum length of 512 characters.
    """
...
    store: Optional[bool]
    """
    Whether or not to store the output of this chat completion request for use in
    our [model distillation](https://platform.openai.com/docs/guides/distillation)
    or [evals](https://platform.openai.com/docs/guides/evals) products.
    """

from: https://github.com/openai/openai-python/blob/7a6517d81e4ae9e9e9527cd401bb76937983dfef/src/openai/types/chat/completion_create_params.py#L126C1-L134C8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant