Fix LLMJudge input handling to preserve BinaryContent as separate message part instead of stringifying #2173
Conversation
| # For non-string inputs (e.g., BinaryContent), build a list | ||
| prompt_parts: list[str | UserContent] = [] | ||
| prompt_parts.append('<Input>\n') | ||
| prompt_parts.append(inputs) |
There was a problem hiding this comment.
inputs may itself be a list of UserContents, in which case we should extend rather than `append
There was a problem hiding this comment.
We should probably make the signature UserContent, Sequence[UserContent] for inputs then as well? Is there a specific reason we are keeping it Any?
| <Rubric> | ||
| {rubric} | ||
| </Rubric> | ||
| """) |
There was a problem hiding this comment.
Similar to below, can we have build a list of strings and conditionally add the item for ExpectedOutput, and join them all at the end? That'll reduce the duplication a bit.
We can likely also merge the 2 isinstance(inputs, str) branches, and only join if isinstance(inputs, str), and return the entire list otherwise
There was a problem hiding this comment.
I've refactored it to reduce duplication
…inary_content_incorrect_parsing_fix
Fix LLMJudge input handling to preserve BinaryContent as separate message part instead of stringifying
Fixes #2089