[bot] Add Llama Stack Client Python SDK integration for inference, agents, and responses execution instrumentation (16,527 weekly downloads) #535

Activity

1. braintrust-bot

   added the

   Feature

   issue type on Jun 29, 2026
2. Abhijeet Prasad (AbhiPrasad)

   added

   new-integration

   on Jul 7, 2026
3. Abhijeet Prasad (AbhiPrasad)

   changed the title [-][bot] Add Llama Stack Client Python SDK integration for inference, agents, and responses execution instrumentation[/-] [+][bot] Add Llama Stack Client Python SDK integration for inference, agents, and responses execution instrumentation (16,527 weekly downloads)[/+] on Aug 10, 2026
4. AbhiPrasad commented on Aug 10, 2026

   

   Abhijeet Prasad (AbhiPrasad)

   on Aug 10, 2026

   Member

   More actions

   Spec compliance review

   Automated review of the requirements stated in this issue against the Braintrust instrumentation specification (braintrust-spec → skills/instrumentation-spec/references/instrumentation-guide.md and references/features/). This reviews the issue text, not any implementation.

   Verdict: NEEDS REVISION — and, more urgently, the API surface this issue specifies is stale and mostly does not exist in the current llama-stack-client.
   Category: completion API + agentic framework + embeddings (mixed; all OpenAI-compatible surfaces)
   Already implemented in this repo: no — there is no py/src/braintrust/integrations/llama_stack/, no wrapper, no patcher, no [tool.braintrust.matrix] entry, and no nox session. A repo grep for llama_stack / llama-stack / llamastack returns zero hits in SDK source. (The llama-index-core entries at py/pyproject.toml:460-462 are a different package and unrelated.)

   ---

   ⚠️ Lead finding: the specified API surface does not exist

   I installed the package to verify this issue's claims rather than taking the tables at face value:

   uv pip install llama-stack-client   # resolved: llama-stack-client==0.7.5
   

   I then enumerated the resource tree at llama_stack_client/resources/ and the client properties in llama_stack_client/_client.py. Results:

   Surface this issue specifies	Exists in llama-stack-client 0.7.5?
   client.inference.chat_completion(...)	❌ No. There is no client.inference resource at all.
   client.inference.completion(...)	❌ No.
   client.inference.embeddings(...)	❌ No.
   client.agents.create(...)	❌ No client.agents resource exists.
   client.agents.sessions.create(...)	❌ No.
   client.agents.turns.create(...)	❌ No — and therefore no AgentTurnResponseStreamChunk, no AgentTurnResponseStepStartPayload / AgentTurnResponseStepCompletePayload.
   client.responses.create(...)	✅ Yes.

   The only inference resource is client.alpha.inference (llama_stack_client/resources/alpha/inference.py), and it exposes exactly one method: rerank().

   The surfaces that actually exist today are:

   Real method	HTTP route	Shape
   client.chat.completions.create()	POST /v1/chat/completions	OpenAI-compatible chat completions
   client.completions.create()	POST /v1/completions	OpenAI-compatible legacy text completions
   client.embeddings.create()	POST /v1/embeddings	OpenAI-compatible embeddings
   client.responses.create()	POST /v1/responses	OpenAI-compatible Responses API
   client.alpha.inference.rerank()	—	rerank (see "Blocked on spec changes")

   Also present and unaddressed by this issue: client.conversations, client.moderations, client.safety, client.shields, client.vector_io, client.vector_stores, client.batches, client.files, client.prompts.

   Two consequences:

   1. The version claim is wrong. The issue says:

      Latest stable release: v0.2.x / 0.6.0 (March 2026)

      That is both internally inconsistent and stale. The current release is 0.7.5. The inference/agents API this issue describes is the pre-0.3 Llama Stack shape, which has since been replaced by OpenAI-compatible routes.

   2. Most of the "What needs to be instrumented" section needs to be rewritten, not merely corrected — the Inference API table and the entire Agents API section describe methods that cannot be patched because they do not exist.

   ---

   Conflicts with the spec

   1. "Separate from the OpenAI SDK shape" is incorrect, and it drives the wrong normalization conclusion

   The issue states:

   Distinct client from OpenAI: LlamaStackClient is a Stainless-generated client (like Groq, Mistral, Together) but with a Llama Stack-specific API shape. It is not a subclass of openai.OpenAI and cannot be wrapped with wrap_openai().

   The class-identity half is true. The payload half is not. In 0.7.5 the payloads are explicitly OpenAI-compatible — the docstring on client.chat.completions.create literally reads "Generate an OpenAI-compatible chat completion for the given messages using the specified model." The request takes OpenAI messages, tools, tool_choice, parallel_tool_calls, response_format, and the response type CompletionCreateResponse (llama_stack_client/types/chat/completion_create_response.py) is:

   • {id, choices[], created, model, usage}
   • choices[].finish_reason: Literal["stop", "length", "tool_calls", "content_filter", "function_call"]
   • choices[].message.tool_calls[].function.arguments: str — already a JSON-encoded string
   • choices[].message.tool_calls[].type: Literal["function"], with id: str

   Spec rule: instrumentation-guide.md §Canonical payload format — Braintrust ships built-in UI normalizers only for OpenAI, Anthropic, and Google. Llama Stack is none of these, so "instrumentation MUST convert payloads into the OpenAI Chat Completions format so that the UI can render them." There is no "preserve native payload" allowance here.

   Correction: State the requirement explicitly — input MUST be an OpenAI-style ordered messages array (§Input capture) and output MUST be an array of OpenAI-style choice objects (§Output capture) — and note that for chat.completions this is effectively a passthrough rather than a bespoke translation layer. Do not let the "distinct client" framing imply that a Llama-Stack-native payload may be logged as-is.

   2. total_tokens proposed as a span metric

   Token usage: ChatCompletionResponse includes a usage field with prompt_tokens, completion_tokens, total_tokens — directly usable for span metrics.

   "Directly usable" is wrong for the third one. total_tokens is not a Braintrust metric key.

   Spec rules:

   • instrumentation-guide.md §Metrics Reference: "SDKs MUST NOT add metrics beyond the keys listed in this guide." The canonical total is tokens.
   • features/token-and-cost-metrics.md §Canonical metrics: "input_tokens, output_tokens, and total_tokens are accepted by some OpenTelemetry ingestion adapters and normalized to prompt_tokens, completion_tokens, and tokens. SDKs emitting Braintrust-native metrics MUST use the canonical Braintrust names directly."

   Correction: map usage.total_tokens → metrics.tokens. (Prior art already does exactly this — see the _TOKEN_NAME_MAP entry "total_tokens": "tokens" in py/src/braintrust/integrations/mistral/tracing.py.)

   3. metadata.provider treated as optional

   The integration should capture the model_id and, where available, the backend provider from response metadata.

   Spec rules: instrumentation-guide.md §Metadata — "Every LLM span MUST include" both model and provider, and "The provider field is required even when model names are globally recognizable, because gateways and resellers can price the same model differently." features/token-and-cost-metrics.md §Span identity marks metadata.provider as MUST.

   Correction: metadata.provider must be emitted unconditionally. A fixed "llama-stack" is the defensible value, since Llama Stack is precisely the gateway whose routing determines which pricing surface applies — this is the case the spec's "provider, gateway, or reseller" wording exists for. Separately, metadata.model SHOULD come from the response's model field, not the request model_id: §Metadata says "The model field SHOULD use the model string from the API response (which may include a version suffix) rather than the string the user passed in the request."

   4. Control-plane calls proposed as instrumentation targets

   client.agents.create(agent_config) | Create an agent with a specific system prompt, tools, and model
   client.agents.sessions.create(agent_id, ...) | Create a session for multi-turn conversations

   These are resource-registration calls, not model calls. instrumentation-guide.md §Span types defines no span type that fits them (llm, function, task, tool, eval, score, automation, preprocessor, classifier, review), and §Data capturing policy states "Instrumentation MUST NOT capture any data or fields unless this guide explicitly requires or allows them."

   Correction: drop them. (Independently moot — the resource does not exist.)

   5. logprobs listed among the response data to capture

   ChatCompletionResponse includes completion_message (content + tool calls), logprobs, and usage metadata

   Spec rule: §Data capturing policy — "Instrumentation MUST NOT capture any data or fields unless this guide explicitly requires or allows them. This applies even when the backend can store arbitrary JSON or a provider/framework SDK exposes additional data." Neither logprobs nor top_logprobs is sanctioned anywhere in the guide.

   Correction: state explicitly that choices[].logprobs is excluded from both output and metadata.

   6. The Responses API is treated as unconditionally agentic

   Responses API (agent execution)

   Spec rule: instrumentation-guide.md §Completion APIs vs Agentic APIs lists "OpenAI Responses API (with tools)" as the agentic example. A responses.create call with no tools runs no internal tool loop and is a single request-response — one llm span, no children.

   Correction: gate the span-tree shape on the presence of tools: with tools → the agentic tree; without tools → exactly one llm span per §One span per API call.

   ---

   Missing requirements

   The issue is written as a discovery/gap report, so it never states most of what the spec mandates. Each of the following must be added.

   1. Span structure — the issue never names a single span type. Required:

   • chat.completions.create, completions.create, embeddings.create → exactly one llm span per API call (§One span per API call: "A streaming call produces a single span that closes when the stream finishes, not one span per chunk.").
   • responses.create with tools → §Agentic API Instrumentation: one parent task span (span_attributes.type = "task") wrapping child llm spans and child tool spans.

   ⚠️ Prior-art tension worth resolving in this issue. The existing OpenAI Responses instrumentation in this repo does not follow the task-parent shape: py/src/braintrust/integrations/openai/tracing.py:1151 and :1206 open the Responses span as SpanTypeAttribute.LLM (name openai.responses.create, tracing.py:1138), and _log_response_tool_spans (tracing.py:1108-1115) attaches flat SpanTypeAttribute.TOOL children to it. So shipped practice is llm + flat tool children, while the spec calls for task + llm/tool children. This issue should state which shape it intends and why, rather than leaving it to the implementer.

   2. Tool-call payload shape. §Tool calls in completion API output requires, for any provider without a UI normalizer: id (MUST), type: "function" (MUST), function.name (MUST), and function.arguments as a JSON-encoded string (MUST) — "not a JSON object … The string MAY be malformed JSON if the model produced invalid output — in which case it MUST be passed through as-is rather than dropped." Free for chat.completions (already str), but Responses-API function_call output items use call_id + arguments and need explicit mapping to the OpenAI tool-call object.

   3. metadata.tools and tool request controls. §Available tool definitions: "Available tool definitions MUST be placed in metadata.tools on every llm span whose request makes one or more tools available", as OpenAI-shaped tool objects; omit entirely rather than emitting []; and "If a model call occurs inside an agentic tool-use loop, each child llm span MUST include the metadata.tools value for the tools available to that specific model call." Also required when supplied: metadata.tool_choice (normalized to "auto" / "none" / "required" / {type:"function",...}), metadata.parallel_tool_calls, metadata.max_tool_calls. All of tools, tool_choice, parallel_tool_calls, max_tool_calls are real parameters on client.responses.create; tools, tool_choice, parallel_tool_calls are real on client.chat.completions.create. Per §Provider-native tool types, the built-in Responses tool types (web_search, file_search, mcp, code_interpreter) MUST keep their provider-native type and MUST NOT be given fake function names or schemas.

   4. A metadata allowlist — this is the largest omission. §Metadata permits only model, provider, and "temperature, top_p, max_tokens, frequency_penalty, presence_penalty, stop, and response_format", plus the tool fields above and metadata.prompt, closing with "Any additional metadata field requires a specification update before SDKs emit it."

   The issue names no allowlist, which leaves an implementer free to emit unsanctioned keys. Concretely not sanctioned and MUST NOT be emitted: agent_id, session_id, turn_id (all three appear in this issue's own method signatures), and the real 0.7.5 request params conversation, previous_response_id, guardrails, max_infer_iters, safety_identifier, prompt_cache_key, service_tier, seed, user, n, logit_bias, logprobs, top_logprobs, reasoning_effort, include, background, store, plus the response envelope fields id, object, created.

   5. time_to_first_token. The issue discusses streaming lifecycle:

   The integration must handle streaming span lifecycle (start span on call, accumulate chunks/events, finalize on exhaustion).

   …which correctly matches §Streaming → Accumulation, but omits the metric. §Streaming → Additional metrics: "This metric MUST be captured for all streaming calls. It is measured by the SDK, not reported by the provider." features/token-and-cost-metrics.md lists it as MUST for streaming spans.

   6. Reasoning and cache metrics. CompletionCreateResponse.Usage carries completion_tokens_details.reasoning_tokens and prompt_tokens_details.cached_tokens. Required mappings:

   • completion_tokens_details.reasoning_tokens → metrics.completion_reasoning_tokens (§Reasoning Models: MUST when the provider reports it)
   • prompt_tokens_details.cached_tokens → metrics.prompt_cached_tokens (§Token Caching: SHOULD when reported)

   Per features/token-and-cost-metrics.md §Canonical metrics: "Omit a metric when the provider did not report it and the SDK cannot compute it accurately. Do not fabricate zero values to hide missing provider data." Note every field on Usage is Optional[int], so omission handling is load-bearing here.

   7. Embeddings are entirely unspecified. The issue lists the surface and says only "Text embeddings generation". features/embeddings.md is prescriptive and none of it appears in this issue:

   • Span model: "Each embedding request MUST produce exactly one llm span, including a request that contains multiple input items. Instrumentation MUST NOT emit one span per returned embedding." Plus metadata.provider and metadata.model.
   • Canonical input: { inputs: [{ content: string | EmbeddingContentPart[] }], output_dimensions?: number }. The real client.embeddings.create has a dimensions param → map to output_dimensions. "Other provider configuration fields MUST NOT be captured until added to this specification" — so encoding_format and user are excluded.
   • Canonical output: { count: number } and nothing else. CreateEmbeddingsResponse.data[].embedding is Union[List[float], str] (float array or base64) and MUST NOT be logged: "Instrumentation MUST NOT capture: raw vector values, prefixes or samples of vector values, vector hashes, vector dimensions, including the actual returned dimension, vector norms or other derived vector statistics." Also "Provider-native embedding responses MUST NOT be logged."
   • Metrics: emit prompt_tokens and tokens only when reported; "completion_tokens … MUST be omitted. Instrumentation MUST NOT fabricate completion_tokens: 0." This directly contradicts the issue's blanket "usage … directly usable for span metrics" line, since CreateEmbeddingsResponse.Usage exposes prompt_tokens and total_tokens.

   8. Error handling. Not mentioned anywhere in the issue. Provider failures MUST populate the span's top-level error field. For embeddings specifically, features/embeddings.md §Errors and partial responses: "If a provider returns partial batch results with an error, count MUST equal the number actually returned", and "instrumentation MUST NOT log provider-native output while reporting an error."

   9. context.span_origin provenance. §Context: "Braintrust-created spans MUST also include span-origin provenance." Repo convention is a module-level instrumentation constant threaded through a local start_span shim — see py/src/braintrust/integrations/mistral/tracing.py:22-29:

   _INSTRUMENTATION = "mistral-auto"
   
   
   def start_span(*args, **kwargs):
       internal = dict(kwargs.get("internal") or {})
       internal.setdefault("instrumentation", _INSTRUMENTATION)
       kwargs["internal"] = internal
       return _bt_start_span(*args, **kwargs)

   The issue should require the equivalent _INSTRUMENTATION = "llama-stack-auto".

   10. Span naming. §Span naming requires names that reflect provider and operation; the issue specifies none. Repo convention is dotted paths (e.g. openai.responses.create at openai/tracing.py:1138), so llama_stack.chat.completions.create and similar.

   11. Multimodal / attachments. Not mentioned. Both chat.completions messages and responses.create input accept media parts — response_create_params.py defines input_image and input_file content types. §Multimodal / Attachments: "SDKs MUST replace the raw media with Braintrust attachment references inside the log row's input or output payload", and "If conversion or upload fails, instrumentation MUST preserve the original payload and MUST NOT throw an exception that prevents span export."

   12. Parent-span aggregation for agentic runs. §Parent span: the task span SHOULD capture the initial input, the final response, and "Aggregated token counts across all child LLM calls (tokens, prompt_tokens, completion_tokens)".

   13. Child-span ordering. §Ordering: "Child spans MUST reflect the actual execution order."

   14. Repo wiring specifics. The issue correctly observes these are absent but does not state them as requirements. Per CLAUDE.md: a [tool.braintrust.matrix.llama-stack-client] block with an explicit latest pin, a [tool.braintrust.cassette-dirs] entry (required for the stale-cassette pre-commit check), a test_llama_stack nox session in py/noxfile.py, inclusion in .github/workflows/checks.yaml, and VCR-backed tests with checked-in cassettes under py/src/braintrust/integrations/llama_stack/cassettes/<version>/ — not mock-based coverage.

   ---

   Blocked on spec changes (braintrustdata/braintrust-spec)

   These surfaces exist in llama-stack-client 0.7.5 but cannot be instrumented conformantly today, because the spec does not define a shape for them. They should be moved out of the "instrument this" list and tracked as spec work.

   1. Reranking — client.alpha.inference.rerank(). This is the one method the real alpha.inference resource exposes, and instrumentation-guide.md §TODO names it as explicitly unspecified: "Reranking APIs — instrumentation for reranking endpoints (e.g. Cohere rerank, Jina Reranker), including input/output structure and relevance score metrics." Needs a spec addition first.

   2. Safety / shields / moderation — client.safety.run_shield(), client.moderations.create(), client.shields.*. No sanctioned span type or payload shape exists. features/classifiers.md is not applicable: it specifies eval-framework classifier functions (EvalClassifier, Classification, the classifiers field on the evaluator interface), not provider-side guardrail/moderation API calls. Any shields metadata would also violate the §Metadata allowlist.

   3. Retrieval / vector stores — client.vector_io.query(), client.vector_stores.search(). The guide specifies tool spans only for model-initiated calls inside an agentic loop (§Child tool spans). A directly user-issued retrieval call has no defined span shape.

   4. metadata.provider for a self-hosted routing gateway. Llama Stack routes to Ollama, vLLM, TGI, Together, Fireworks, SambaNova, and Meta endpoints, and the OpenAI-compatible response does not reliably expose which backend served the request. features/token-and-cost-metrics.md frames provider as the pricing surface, but gives no guidance for local/self-hosted inference where no pricing exists at all. A short clarification would let this integration pick a value defensibly instead of guessing.

   5. Strict metadata allowlist vs. shipped practice. Flagging for consistency, not to license a violation here: existing integrations already emit keys outside the §Metadata allowlist (e.g. agent_id, guardrails, safe_prompt and the response id/object/created in the Mistral integration; status, service_tier, background in the OpenAI Responses path), and integrations/utils.py emits a duration metric that is not in the §Metrics Reference table. Either the spec's allowlist needs to widen or this issue should not treat prior art as precedent for extra metadata.

   ---

   Suggested edits to this issue

   • Correct the version claim from "v0.2.x / 0.6.0 (March 2026)" to 0.7.5, and note the verification method.
   • Delete the Agents API section entirely and delete the client.inference.* table; replace with the real surfaces: client.chat.completions.create, client.completions.create, client.embeddings.create, client.responses.create, plus the AsyncLlamaStackClient equivalents.
   • Rewrite the "Distinct client from OpenAI" note: keep "not an openai.OpenAI subclass, so wrap_openai() does not apply", but replace "Llama Stack-specific API shape" with "OpenAI-compatible payloads on /v1/chat/completions, /v1/embeddings, and /v1/responses", and add that conversion to OpenAI Chat Completions format is nonetheless required (no UI normalizer exists for Llama Stack) and is effectively a passthrough.
   • Add an explicit span-structure section: chat.completions / completions / embeddings → one llm span; responses.create with tools → task parent + child llm/tool spans; responses.create without tools → one llm span. Resolve the task-vs-llm parent question against the existing OpenAI Responses behavior noted above.
   • Add a metadata allowlist section listing only the sanctioned keys, plus an explicit "do NOT emit" list: agent_id, session_id, turn_id, conversation, previous_response_id, guardrails, max_infer_iters, safety_identifier, prompt_cache_key, service_tier, seed, user, n, logit_bias, logprobs, top_logprobs, reasoning_effort, include, background.
   • Replace "usage … directly usable for span metrics" with the explicit mapping table: prompt_tokens→prompt_tokens, completion_tokens→completion_tokens, total_tokens→**tokens**, completion_tokens_details.reasoning_tokens→completion_reasoning_tokens, prompt_tokens_details.cached_tokens→prompt_cached_tokens; omit anything the provider did not report, never fabricate zeros.
   • Make metadata.provider unconditional ("llama-stack") and source metadata.model from the response, not the request.
   • Add time_to_first_token to the streaming requirement.
   • Add a dedicated embeddings section citing features/embeddings.md: one llm span, canonical {inputs, output_dimensions} input (map the dimensions param), {count}-only output, no vectors/dimensions/norms, no completion_tokens.
   • Add requirements for: metadata.tools / tool_choice / parallel_tool_calls / max_tool_calls; JSON-string tool-call arguments; top-level error on failure; attachment conversion for input_image / input_file parts; context.span_origin via _INSTRUMENTATION = "llama-stack-auto"; span naming; child-span ordering; parent-span token aggregation.
   • Add the repo wiring checklist: matrix pin, [tool.braintrust.cassette-dirs] entry, test_llama_stack nox session, CI workflow inclusion, and VCR cassettes (not mocks).
   • Move rerank, safety/shields/moderations, and vector-store surfaces into an explicit "blocked on braintrust-spec" section rather than listing them as instrumentable work.

   ---

   _{No files were modified, no labels changed, and no PR was opened as part of this review.}
5. AbhiPrasad commented on Oct 14, 2026

   

   Abhijeet Prasad (AbhiPrasad)

   on Oct 14, 2026

   Member

   More actions

   Closing as not planned. The inference / agents APIs described here no longer exist in current llama-stack-client releases, payloads are now OpenAI-compatible, and adoption is low (~9.5k weekly downloads).
6. Abhijeet Prasad (AbhiPrasad)

   closed this as not plannedon Oct 14, 2026