Repository navigation
feat(evaluations): Set JSON as standard for output to pass to judge - #203
Merged
Merged
Conversation
jeffdupont
approved these changes
Oct 22, 2026
donei003
approved these changes
Oct 22, 2026
Merged
jeffdupont
pushed a commit
that referenced
this pull request
Oct 23, 2026
🤖 I have created a release *beep* *boop* --- <details><summary>launchdarkly-ai-server: 0.4.0</summary> ## [0.4.0](launchdarkly-ai-server-0.3.0...launchdarkly-ai-server-0.4.0) (2026-10-23) ### ⚠ BREAKING CHANGES * **evaluations:** keep the management API client internal ([#202](#202)) * **evaluations:** rename base uri to match spec ([#199](#199)) * **evaluations:** add test proving harness edge case matches spec ([#200](#200)) * **evaluations:** route handlers by provider and mode * **evaluations:** mark evaluations from code as experimental ([#157](#157)) ### Experimental * **evaluations:** add test proving harness edge case matches spec ([#200](#200)) ([fa7162e](fa7162e)) * **evaluations:** keep the management API client internal ([#202](#202)) ([17270ed](17270ed)) * **evaluations:** mark evaluations from code as experimental ([#157](#157)) ([1389765](1389765)) * **evaluations:** rename base uri to match spec ([#199](#199)) ([7d93f9c](7d93f9c)) * **evaluations:** route handlers by provider and mode ([f62591f](f62591f)) * **evaluations:** Set JSON as standard for output to pass to judge ([#203](#203)) ([ffc728b](ffc728b)) * **skills:** Correct the HTTP 403 advice now that server-side FDv2 is available to every account ([#178](#178)) ([b17e322](b17e322)) ### Features * **agui:** add a shared reader for handler streams ([#164](#164)) ([fc08dd5](fc08dd5)) * **AIC-3534:** add a TypeSafe Jev judge handler ([#149](#149)) ([1b41ea4](1b41ea4)) * **client:** add configure for SDK defaults ([#158](#158)) ([3736767](3736767)) * **langchain:** emit AG-UI text and tool events ([#163](#163)) ([ab5e9e1](ab5e9e1)) * **openai:** emit AG-UI text and tool events ([#161](#161)) ([2b1b2d4](2b1b2d4)) ### Bug Fixes * **AIC-3570:** re-resolve graph flags on every invoke and stream ([#187](#187)) ([8fdc4d4](8fdc4d4)) * **AIC-3571:** visit every child before its parent in reverse_traverse ([#189](#189)) ([87c73d9](87c73d9)) * **AIC-3572:** treat a variation without _ldMeta.enabled as disabled ([#184](#184)) ([7a0d260](7a0d260)) * **AIC-3574:** build the LaunchDarkly client off the event loop ([#185](#185)) ([bf03b6c](bf03b6c)) * **AIC-3576:** record deferred judge scores under the evaluated config ([#190](#190)) ([354d3f1](354d3f1)) * **AIC-3577:** send object tool results to the model as JSON ([#188](#188)) ([dead4bd](dead4bd)) * **AIC-3596:** name the [otel] extra in the missing-OTel warning ([#186](#186)) ([90ce7bf](90ce7bf)) * **client:** match a handler's provides_for after normalizing its mode ([f62591f](f62591f)) * give native graph spans their LaunchDarkly identity ([#121](#121)) ([e863b6d](e863b6d)) </details> <details><summary>launchdarkly-ai-python: 0.3.0</summary> ## [0.3.0](launchdarkly-ai-python-0.2.0...launchdarkly-ai-python-0.3.0) (2026-10-23) ### ⚠ BREAKING CHANGES * **evaluations:** mark evaluations from code as experimental ([#157](#157)) ### Experimental * **evaluations:** mark evaluations from code as experimental ([#157](#157)) ([1389765](1389765)) </details> <details><summary>launchdarkly-ai-claude-agents: 0.2.6</summary> ## [0.2.6](launchdarkly-ai-claude-agents-0.2.5...launchdarkly-ai-claude-agents-0.2.6) (2026-10-23) ### Features * **claude:** emit AG-UI text and tool events ([#165](#165)) ([3f3a0ce](3f3a0ce)) * **client:** add configure for SDK defaults ([#158](#158)) ([3736767](3736767)) ### Bug Fixes * **AIC-3571:** visit every child before its parent in reverse_traverse ([#189](#189)) ([87c73d9](87c73d9)) * **AIC-3577:** send object tool results to the model as JSON ([#188](#188)) ([dead4bd](dead4bd)) * **claude-agents:** fail the run when the CLI reports an API error as success ([#174](#174)) ([3d1a768](3d1a768)) * give native graph spans their LaunchDarkly identity ([#121](#121)) ([e863b6d](e863b6d)) </details> <details><summary>launchdarkly-ai-claude-messages: 0.2.5</summary> ## [0.2.5](launchdarkly-ai-claude-messages-0.2.4...launchdarkly-ai-claude-messages-0.2.5) (2026-10-23) ### Features * **claude:** emit AG-UI text and tool events ([#165](#165)) ([3f3a0ce](3f3a0ce)) * **client:** add configure for SDK defaults ([#158](#158)) ([3736767](3736767)) ### Bug Fixes * **AIC-3577:** send object tool results to the model as JSON ([#188](#188)) ([dead4bd](dead4bd)) </details> <details><summary>launchdarkly-ai-openai-agents: 0.2.6</summary> ## [0.2.6](launchdarkly-ai-openai-agents-0.2.5...launchdarkly-ai-openai-agents-0.2.6) (2026-10-23) ### Features * **client:** add configure for SDK defaults ([#158](#158)) ([3736767](3736767)) * **openai:** emit AG-UI text and tool events ([#161](#161)) ([2b1b2d4](2b1b2d4)) ### Bug Fixes * **AIC-3571:** visit every child before its parent in reverse_traverse ([#189](#189)) ([87c73d9](87c73d9)) * **AIC-3577:** send object tool results to the model as JSON ([#188](#188)) ([dead4bd](dead4bd)) * give native graph spans their LaunchDarkly identity ([#121](#121)) ([e863b6d](e863b6d)) </details> <details><summary>launchdarkly-ai-vercel-agents: 0.1.1</summary> ## [0.1.1](launchdarkly-ai-vercel-agents-0.1.0...launchdarkly-ai-vercel-agents-0.1.1) (2026-10-23) ### Features * **AIC-3408:** add Vercel AI SDK messages and agents adapters ([#103](#103)) ([64d679e](64d679e)) </details> <details><summary>launchdarkly-ai-litellm-agents: 0.1.1</summary> ## [0.1.1](launchdarkly-ai-litellm-agents-0.1.0...launchdarkly-ai-litellm-agents-0.1.1) (2026-10-23) ### Features * add LiteLLM messages and agents integrations ([#102](#102)) ([f2e9886](f2e9886)) </details> <details><summary>launchdarkly-ai-openai-messages: 0.2.5</summary> ## [0.2.5](launchdarkly-ai-openai-messages-0.2.4...launchdarkly-ai-openai-messages-0.2.5) (2026-10-23) ### Features * **client:** add configure for SDK defaults ([#158](#158)) ([3736767](3736767)) * **openai:** emit AG-UI text and tool events ([#161](#161)) ([2b1b2d4](2b1b2d4)) ### Bug Fixes * **AIC-3577:** send object tool results to the model as JSON ([#188](#188)) ([dead4bd](dead4bd)) </details> <details><summary>launchdarkly-ai-vercel-messages: 0.1.1</summary> ## [0.1.1](launchdarkly-ai-vercel-messages-0.1.0...launchdarkly-ai-vercel-messages-0.1.1) (2026-10-23) ### Features * **AIC-3408:** add Vercel AI SDK messages and agents adapters ([#103](#103)) ([64d679e](64d679e)) </details> <details><summary>launchdarkly-ai-litellm-messages: 0.1.1</summary> ## [0.1.1](launchdarkly-ai-litellm-messages-0.1.0...launchdarkly-ai-litellm-messages-0.1.1) (2026-10-23) ### Features * add LiteLLM messages and agents integrations ([#102](#102)) ([f2e9886](f2e9886)) </details> <details><summary>launchdarkly-ai-langchain-agents: 0.2.6</summary> ## [0.2.6](launchdarkly-ai-langchain-agents-0.2.5...launchdarkly-ai-langchain-agents-0.2.6) (2026-10-23) ### Features * **client:** add configure for SDK defaults ([#158](#158)) ([3736767](3736767)) * **langchain:** emit AG-UI text and tool events ([#163](#163)) ([ab5e9e1](ab5e9e1)) ### Bug Fixes * **AIC-3577:** send object tool results to the model as JSON ([#188](#188)) ([dead4bd](dead4bd)) * give native graph spans their LaunchDarkly identity ([#121](#121)) ([e863b6d](e863b6d)) </details> <details><summary>launchdarkly-ai-langchain-messages: 0.2.5</summary> ## [0.2.5](launchdarkly-ai-langchain-messages-0.2.4...launchdarkly-ai-langchain-messages-0.2.5) (2026-10-23) ### Features * **client:** add configure for SDK defaults ([#158](#158)) ([3736767](3736767)) * **langchain:** emit AG-UI text and tool events ([#163](#163)) ([ab5e9e1](ab5e9e1)) ### Bug Fixes * **AIC-3577:** send object tool results to the model as JSON ([#188](#188)) ([dead4bd](dead4bd)) </details> <details><summary>launchdarkly-ai-opencode-agents: 0.2.0</summary> ## [0.2.0](launchdarkly-ai-opencode-agents-0.1.0...launchdarkly-ai-opencode-agents-0.2.0) (2026-10-23) ### ⚠ BREAKING CHANGES * **opencode:** start and stop opencode serve in the handler ([#198](#198)) ### Features * **opencode-agents:** add an OpenCode handler ([#159](#159)) ([fc08e8c](fc08e8c)) * **opencode-agents:** emit the AG-UI text and tool subset ([#162](#162)) ([7677621](7677621)) * **opencode-agents:** open one OpenCode session per call ([#179](#179)) ([b225113](b225113)) * **opencode-agents:** render agent instruction templates ([#182](#182)) ([83175aa](83175aa)) * **opencode:** start and stop opencode serve in the handler ([#198](#198)) ([0748dcf](0748dcf)) ### Bug Fixes * **opencode-agents:** own the HTTP client ([#194](#194)) ([ed2d419](ed2d419)) * **opencode-agents:** skip judges and record structured history ([#177](#177)) ([6fadf1d](6fadf1d)) </details> <details><summary>launchdarkly-ai-typesafe: 0.1.1</summary> ## [0.1.1](launchdarkly-ai-typesafe-0.1.0...launchdarkly-ai-typesafe-0.1.1) (2026-10-23) ### Features * **AIC-3534:** add a TypeSafe Jev judge handler ([#149](#149)) ([1b41ea4](1b41ea4)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Python and JS SDKs had a subtle disagreement in how outputs were formatted to send to judges. Making the decision to spec these outputs as json objects, and adjusting the code to match more closely.
Note
Overview
Aligns how generation output is shown to judges when it is not a plain string:
response_to_evaluateand the output segment ofmessage_historynow go throughtool_result_string, so dict/structured results become JSON text (same rule as tool results) instead of Python’s default string form.Offline evaluation wiring is updated so judge handlers receive the formatted
response_to_evaluatevariable rather than the raw rowoutput, keeping the judgeuser_inputargument consistent with template variables.Tests cover end-to-end eval runs and
build_message_history(including non-ASCII and empty dict output).Reviewed by Cursor Bugbot for commit f36a14b. Bugbot is set up for automated code reviews on this repo. Configure here.