add APIs for llm obs #8135

gary-huang · 2024-12-30T15:27:59Z

What Does This Do

Add APIs for LLM obs manual instrumentation
Starting spans

LLMObsSpan span = LLMObs.startLLMSpan(String spanName, String modelName, String modelProvider, @Nullable String mlApp, @Nullable String sessionID)
LLMObsSpan span = LLMObs.startAgentSpan(String spanName, @Nullable String mlApp, @Nullable String sessionID)
LLMObsSpan span = LLMObs.startToolSpan(String spanName, @Nullable String mlApp, @Nullable String sessionID)
LLMObsSpan span = LLMObs.startTaskSpan(String spanName, @Nullable String mlApp, @Nullable String sessionID)
LLMObsSpan span = LLMObs.startWorkflowSpan(String spanName, @Nullable String mlApp, @Nullable String sessionID)

Interacting with spans

span.annotateIO(List<Map<String, Object>> inputData, List<Map<String, Object>> outputData)
span.annotateIO(String inputData, String outputData)
span.setMetadata(Map<String, Object> metadata)
span.setMetrics(Map<String, Number> metrics)
span.setMetric(CharSequence key, int value)
setMetric(CharSequence key, long value)
setMetric(CharSequence key, double value)
span.setTags(Map<String, Object> tags)
span.setTag(String key, String value)
span.setTag(String key, boolean value)
span.setTag(String key, int value)
span.setTag(String key, long value)
span.setTag(String key, double value)
span.setError(boolean error)
span.setErrorMessage(String errorMessage)
span.addThrowable(Throwable throwable)
span.finish()

A sample spring controller using the APIs above

package com.example.restservice;

import datadog.trace.api.appsec.LoginEventCallback;
import datadog.trace.api.civisibility.telemetry.tag.AgentlessLogSubmissionEnabled;
import java.lang.management.ManagementFactory;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.concurrent.atomic.AtomicLong;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

import datadog.trace.api.llmobs.LLMObs;

@RestController
public class GreetingController {

	private static final String template = "Hello, %s, the %dth visitor!";
	private final AtomicLong counter = new AtomicLong();

	private static final Logger logger = LoggerFactory.getLogger(GreetingController.class);

	private String getCurrentSessionID() {
		return ManagementFactory.getRuntimeMXBean().getStartTime() + "-" + counter.get();
	}

	private long taskCall(String input) {
		var currentCount = counter.get();
		Throwable t = null;
		var taskSpan = LLMObs.startTaskSpan("divide_by_zero_task", null, getCurrentSessionID());
		try {
			var l = 100 / 0;
			logger.info("TASK COMPLETE");
		} catch (Exception e) {
			t = e;
		}
		taskSpan.addThrowable(t);
		taskSpan.annotateIO("param input is " + input, "current count is " + currentCount);
		taskSpan.finish();
		logger.info("taskSpan {}", taskSpan);
		return currentCount;
	}

	@GetMapping("/greeting")
	public Greeting greeting(@RequestParam(value = "name", defaultValue = "World") String name) {
		var session = getCurrentSessionID();

		var workflowSpan = LLMObs.startWorkflowSpan("workflow",  null, session);
		var currentCount = taskCall(name);

		var llmObsSpan1 = LLMObs.startLLMSpan("llm1", "gpt4", "openai", "", session);


		var erredLLMSpan = LLMObs.startLLMSpan("llm2-err", "gpt4", "openai", "test-gary-mlapp-override-app", session);


		workflowSpan.setTags(new HashMap<String, Object>(){{put("tag1", 1); put("tag2", "a");}});
		workflowSpan.setMetrics(new HashMap<>(){{put("input_tokens", 100); put("output_tokens", 100l); put("log_prob", 0.1);}});

		List inputList = new ArrayList<>(){
			{
				new HashMap<String, String>() {{
					put("role", "user:" + name);
					put("content", "input");
				}};
				{
					{
						new HashMap<String, String>() {{
							put("role", "system");
							put("content", "system prompt");
						}};
					}
				}
			}};
		List outputList = new ArrayList<>(){{
			new HashMap<String, String>(){{put("role", "assistant"); put("content", "output");}};}};
		llmObsSpan1.annotateIO(inputList, outputList);
		llmObsSpan1.setTag("tag1", 1);
		llmObsSpan1.setMetadata(new HashMap<String, Object>(){{put("temperature", "1"); put("max_tokens", 100);}});

		erredLLMSpan.annotateIO("span2 in", "span2 out");
		erredLLMSpan.setTags(new HashMap<String, Object>(){{put("tag1", 1); put("tag2", "a");}});
		erredLLMSpan.setErrorMessage("aaa");

		var resp = String.format(template, name, currentCount);
		workflowSpan.annotateIO(name, resp);

		logger.info("workflow {}", workflowSpan);
		logger.info("llm1 {}", llmObsSpan1);
		logger.info("erredLLMSpan {}", erredLLMSpan);

		workflowSpan.finish();
		llmObsSpan1.finish();
		erredLLMSpan.finish();
		return new Greeting(counter.incrementAndGet(), resp);

	}
}

logging for the spans above

2025-02-13T02:52:24.583-05:00  INFO 53094 --- [nio-8080-exec-1] c.e.restservice.GreetingController       : taskSpan datadog.trace.llmobs.domain.DDLLMObsSpan@7b9b90a2, trace_id=479453473746284206, span_id=2948307615785206470, ml_app=gary-test, service=gary-test, span_kind=task, context=DDSpan [ t_id=479453473746284206, s_id=2948307615785206470, p_id=3544560763018401567 ] trace=gary-test/divide_by_zero_task/divide_by_zero_task *errored* tags={_llmobs_tag.input=param input is gary, _llmobs_tag.ml_app=gary-test, _llmobs_tag.output=current count is 0, _llmobs_tag.session_id=1739433141606-0, env=staging, error.message=/ by zero, error.stack=java.lang.ArithmeticException: / by zero
	at com.example.restservice.GreetingController.taskCall(GreetingController.java:36)
	at com.example.restservice.GreetingController.greeting(GreetingController.java:53)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:255)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:188)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:926)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:831)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:564)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:195)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at datadog.trace.instrumentation.springweb6.HandlerMappingResourceNameFilter.doFilterInternal(HandlerMappingResourceNameFilter.java:50)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at org.springframework.web.filter.ServletRequestPathFilter.doFilter(ServletRequestPathFilter.java:52)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:352)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:268)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:389)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:896)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
	at java.base/java.lang.Thread.run(Thread.java:842)
, error.type=java.lang.ArithmeticException, span.kind=task, thread.id=40, thread.name=http-nio-8080-exec-1, version=1.0}
2025-02-13T02:52:24.589-05:00  INFO 53094 --- [nio-8080-exec-1] c.e.restservice.GreetingController       : workflow datadog.trace.llmobs.domain.DDLLMObsSpan@375bdc24, trace_id=479453473746284206, span_id=7264160562704185531, ml_app=gary-test, service=gary-test, span_kind=workflow, context=DDSpan [ t_id=479453473746284206, s_id=7264160562704185531, p_id=3544560763018401567 ] trace=gary-test/workflow/workflow tags={_llmobs_metric.input_tokens=100.0, _llmobs_metric.log_prob=0.1, _llmobs_metric.output_tokens=100.0, _llmobs_tag.input=gary, _llmobs_tag.ml_app=gary-test, _llmobs_tag.output=Hello, gary!, _llmobs_tag.session_id=1739433141606-0, _llmobs_tag.tag1=1, _llmobs_tag.tag2=a, env=staging, span.kind=workflow, thread.id=40, thread.name=http-nio-8080-exec-1, version=1.0}
2025-02-13T02:52:24.589-05:00  INFO 53094 --- [nio-8080-exec-1] c.e.restservice.GreetingController       : llm1 datadog.trace.llmobs.domain.DDLLMObsSpan@65e7b713, trace_id=479453473746284206, span_id=5473581304479074833, ml_app=gary-test, service=gary-test, span_kind=llm, context=DDSpan [ t_id=479453473746284206, s_id=5473581304479074833, p_id=3544560763018401567 ] trace=gary-test/llm1/llm1 tags={_llmobs_tag._llmobs_tag.model_name=gpt4, _llmobs_tag._llmobs_tag.model_provider=openai, _llmobs_tag.metadata={max_tokens=100, temperature=1}, _llmobs_tag.ml_app=gary-test, _llmobs_tag.session_id=1739433141606-0, _llmobs_tag.tag1=1, env=staging, span.kind=llm, thread.id=40, thread.name=http-nio-8080-exec-1, version=1.0}
2025-02-13T02:52:24.589-05:00  INFO 53094 --- [nio-8080-exec-1] c.e.restservice.GreetingController       : erredLLMSpan datadog.trace.llmobs.domain.DDLLMObsSpan@d9d015b, trace_id=479453473746284206, span_id=4239354304348392454, ml_app=test-gary-mlapp-override-app, service=gary-test, span_kind=llm, context=DDSpan [ t_id=479453473746284206, s_id=4239354304348392454, p_id=3544560763018401567 ] trace=gary-test/llm2-err/llm2-err *errored* tags={_llmobs_tag._llmobs_tag.model_name=gpt4, _llmobs_tag._llmobs_tag.model_provider=openai, _llmobs_tag.input=span2 in, _llmobs_tag.ml_app=test-gary-mlapp-override-app, _llmobs_tag.output=span2 out, _llmobs_tag.session_id=1739433141606-0, _llmobs_tag.tag1=1, _llmobs_tag.tag2=a, env=staging, error.message=aaa, span.kind=llm, thread.id=40, thread.name=http-nio-8080-exec-1, version=1.0}
[dd.trace 2025-02-13 02:52:24:589 -0500] [http-nio-8080-exec-1] DEBUG datadog.trace.agent.core.DDSpan - Finished span (PENDING): DDSpan [ t_id=479453473746284206, s_id=7264160562704185531, p_id=3544560763018401567 ] trace=gary-test/workflow/workflow tags={_llmobs_metric.input_tokens=100.0, _llmobs_metric.log_prob=0.1, _llmobs_metric.output_tokens=100.0, _llmobs_tag.input=gary, _llmobs_tag.ml_app=gary-test, _llmobs_tag.output=Hello, gary!, _llmobs_tag.session_id=1739433141606-0, _llmobs_tag.tag1=1, _llmobs_tag.tag2=a, env=staging, span.kind=workflow, thread.id=40, thread.name=http-nio-8080-exec-1, version=1.0}, duration_ns=6872375, forceKeep=false, links=[]

Motivation

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

pr-commenter · 2024-12-30T17:22:05Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	gary/add-llm-obs-api
git_commit_date	1742922414	1742951641
git_commit_sha	`05ccd9a`	`d16b87d`
release_version	1.48.0-SNAPSHOT~05ccd9aa10	1.48.0-SNAPSHOT~d16b87deb5

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1742954693	1742954693
ci_job_id	864930323	864930323
ci_pipeline_id	59986248	59986248
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-2uo-edd9-project-304-concurrent-0-zu02bidk 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-2uo-edd9-project-304-concurrent-0-zu02bidk 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 66 metrics, 5 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.48.0-SNAPSHOT~d16b87deb5, baseline=1.48.0-SNAPSHOT~05ccd9aa10

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.052 s) : 0, 1051513
Total [baseline] (8.721 s) : 0, 8720581
Agent [candidate] (1.071 s) : 0, 1071204
Total [candidate] (8.742 s) : 0, 8741961
section iast
Agent [baseline] (1.178 s) : 0, 1178456
Total [baseline] (9.24 s) : 0, 9240405
Agent [candidate] (1.179 s) : 0, 1178608
Total [candidate] (9.272 s) : 0, 9271622
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.177 s) : 0, 1176771
Total [baseline] (9.247 s) : 0, 9247310
Agent [candidate] (1.18 s) : 0, 1179669
Total [candidate] (9.221 s) : 0, 9221226
section iast_TELEMETRY_OFF
Agent [baseline] (1.172 s) : 0, 1172184
Total [baseline] (9.273 s) : 0, 9273048
Agent [candidate] (1.182 s) : 0, 1181691
Total [candidate] (9.27 s) : 0, 9270396

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.052 s	-
Agent	iast	1.178 s	126.943 ms (12.1%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.177 s	125.258 ms (11.9%)
Agent	iast_TELEMETRY_OFF	1.172 s	120.671 ms (11.5%)
Total	tracing	8.721 s	-
Total	iast	9.24 s	519.824 ms (6.0%)
Total	iast_HARDCODED_SECRET_DISABLED	9.247 s	526.728 ms (6.0%)
Total	iast_TELEMETRY_OFF	9.273 s	552.467 ms (6.3%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.071 s	-
Agent	iast	1.179 s	107.404 ms (10.0%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.18 s	108.464 ms (10.1%)
Agent	iast_TELEMETRY_OFF	1.182 s	110.487 ms (10.3%)
Total	tracing	8.742 s	-
Total	iast	9.272 s	529.662 ms (6.1%)
Total	iast_HARDCODED_SECRET_DISABLED	9.221 s	479.265 ms (5.5%)
Total	iast_TELEMETRY_OFF	9.27 s	528.436 ms (6.0%)

gantt
    title insecure-bank - break down per module: candidate=1.48.0-SNAPSHOT~d16b87deb5, baseline=1.48.0-SNAPSHOT~05ccd9aa10

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (721.674 ms) : 0, 721674
BytebuddyAgent [candidate] (733.745 ms) : 0, 733745
GlobalTracer [baseline] (239.842 ms) : 0, 239842
GlobalTracer [candidate] (244.36 ms) : 0, 244360
AppSec [baseline] (54.508 ms) : 0, 54508
AppSec [candidate] (55.63 ms) : 0, 55630
Debugger [baseline] (5.115 ms) : 0, 5115
Debugger [candidate] (4.5 ms) : 0, 4500
Remote Config [baseline] (709.147 µs) : 0, 709
Remote Config [candidate] (739.682 µs) : 0, 740
Telemetry [baseline] (13.691 ms) : 0, 13691
Telemetry [candidate] (15.991 ms) : 0, 15991
section iast
BytebuddyAgent [baseline] (839.899 ms) : 0, 839899
BytebuddyAgent [candidate] (839.641 ms) : 0, 839641
GlobalTracer [baseline] (230.077 ms) : 0, 230077
GlobalTracer [candidate] (230.491 ms) : 0, 230491
IAST [baseline] (22.869 ms) : 0, 22869
IAST [candidate] (23.11 ms) : 0, 23110
AppSec [baseline] (56.062 ms) : 0, 56062
AppSec [candidate] (55.893 ms) : 0, 55893
Debugger [baseline] (4.158 ms) : 0, 4158
Debugger [candidate] (4.15 ms) : 0, 4150
Remote Config [baseline] (609.957 µs) : 0, 610
Remote Config [candidate] (601.425 µs) : 0, 601
Telemetry [baseline] (8.81 ms) : 0, 8810
Telemetry [candidate] (8.702 ms) : 0, 8702
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (838.146 ms) : 0, 838146
BytebuddyAgent [candidate] (840.48 ms) : 0, 840480
GlobalTracer [baseline] (229.951 ms) : 0, 229951
GlobalTracer [candidate] (230.833 ms) : 0, 230833
IAST [baseline] (22.958 ms) : 0, 22958
IAST [candidate] (22.887 ms) : 0, 22887
AppSec [baseline] (56.136 ms) : 0, 56136
AppSec [candidate] (55.855 ms) : 0, 55855
Debugger [baseline] (4.165 ms) : 0, 4165
Debugger [candidate] (4.198 ms) : 0, 4198
Remote Config [baseline] (596.05 µs) : 0, 596
Remote Config [candidate] (612.685 µs) : 0, 613
Telemetry [baseline] (8.809 ms) : 0, 8809
Telemetry [candidate] (8.83 ms) : 0, 8830
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (834.826 ms) : 0, 834826
BytebuddyAgent [candidate] (843.72 ms) : 0, 843720
GlobalTracer [baseline] (229.897 ms) : 0, 229897
GlobalTracer [candidate] (230.387 ms) : 0, 230387
IAST [baseline] (22.291 ms) : 0, 22291
IAST [candidate] (22.182 ms) : 0, 22182
AppSec [baseline] (55.825 ms) : 0, 55825
AppSec [candidate] (56.071 ms) : 0, 56071
Debugger [baseline] (4.134 ms) : 0, 4134
Debugger [candidate] (4.112 ms) : 0, 4112
Remote Config [baseline] (620.625 µs) : 0, 621
Remote Config [candidate] (604.014 µs) : 0, 604
Telemetry [baseline] (8.649 ms) : 0, 8649
Telemetry [candidate] (8.578 ms) : 0, 8578

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.48.0-SNAPSHOT~d16b87deb5, baseline=1.48.0-SNAPSHOT~05ccd9aa10

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.049 s) : 0, 1049062
Total [baseline] (10.544 s) : 0, 10544152
Agent [candidate] (1.061 s) : 0, 1061148
Total [candidate] (10.519 s) : 0, 10518871
section appsec
Agent [baseline] (1.191 s) : 0, 1190783
Total [baseline] (10.784 s) : 0, 10783879
Agent [candidate] (1.191 s) : 0, 1191458
Total [candidate] (10.791 s) : 0, 10791108
section iast
Agent [baseline] (1.189 s) : 0, 1189280
Total [baseline] (11.046 s) : 0, 11045762
Agent [candidate] (1.191 s) : 0, 1191023
Total [candidate] (11.045 s) : 0, 11044998
section profiling
Agent [baseline] (1.272 s) : 0, 1272173
Total [baseline] (10.856 s) : 0, 10855954
Agent [candidate] (1.274 s) : 0, 1273561
Total [candidate] (10.86 s) : 0, 10860286

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.049 s	-
Agent	appsec	1.191 s	141.721 ms (13.5%)
Agent	iast	1.189 s	140.218 ms (13.4%)
Agent	profiling	1.272 s	223.111 ms (21.3%)
Total	tracing	10.544 s	-
Total	appsec	10.784 s	239.727 ms (2.3%)
Total	iast	11.046 s	501.61 ms (4.8%)
Total	profiling	10.856 s	311.802 ms (3.0%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.061 s	-
Agent	appsec	1.191 s	130.31 ms (12.3%)
Agent	iast	1.191 s	129.875 ms (12.2%)
Agent	profiling	1.274 s	212.413 ms (20.0%)
Total	tracing	10.519 s	-
Total	appsec	10.791 s	272.237 ms (2.6%)
Total	iast	11.045 s	526.127 ms (5.0%)
Total	profiling	10.86 s	341.415 ms (3.2%)

gantt
    title petclinic - break down per module: candidate=1.48.0-SNAPSHOT~d16b87deb5, baseline=1.48.0-SNAPSHOT~05ccd9aa10

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (720.153 ms) : 0, 720153
BytebuddyAgent [candidate] (727.664 ms) : 0, 727664
GlobalTracer [baseline] (240.522 ms) : 0, 240522
GlobalTracer [candidate] (242.604 ms) : 0, 242604
AppSec [baseline] (54.917 ms) : 0, 54917
AppSec [candidate] (55.139 ms) : 0, 55139
Debugger [baseline] (4.445 ms) : 0, 4445
Debugger [candidate] (4.45 ms) : 0, 4450
Remote Config [baseline] (723.285 µs) : 0, 723
Remote Config [candidate] (732.578 µs) : 0, 733
Telemetry [baseline] (12.29 ms) : 0, 12290
Telemetry [candidate] (14.305 ms) : 0, 14305
section appsec
BytebuddyAgent [baseline] (737.72 ms) : 0, 737720
BytebuddyAgent [candidate] (737.859 ms) : 0, 737859
GlobalTracer [baseline] (236.759 ms) : 0, 236759
GlobalTracer [candidate] (236.677 ms) : 0, 236677
AppSec [baseline] (176.438 ms) : 0, 176438
AppSec [candidate] (176.633 ms) : 0, 176633
Debugger [baseline] (4.293 ms) : 0, 4293
Debugger [candidate] (4.322 ms) : 0, 4322
Remote Config [baseline] (649.599 µs) : 0, 650
Remote Config [candidate] (645.135 µs) : 0, 645
Telemetry [baseline] (8.187 ms) : 0, 8187
Telemetry [candidate] (8.612 ms) : 0, 8612
IAST [baseline] (21.533 ms) : 0, 21533
IAST [candidate] (21.364 ms) : 0, 21364
section iast
BytebuddyAgent [baseline] (847.705 ms) : 0, 847705
BytebuddyAgent [candidate] (849.313 ms) : 0, 849313
GlobalTracer [baseline] (232.254 ms) : 0, 232254
GlobalTracer [candidate] (232.213 ms) : 0, 232213
AppSec [baseline] (56.302 ms) : 0, 56302
AppSec [candidate] (56.526 ms) : 0, 56526
Debugger [baseline] (4.199 ms) : 0, 4199
Debugger [candidate] (4.199 ms) : 0, 4199
Remote Config [baseline] (618.875 µs) : 0, 619
Remote Config [candidate] (611.31 µs) : 0, 611
Telemetry [baseline] (8.935 ms) : 0, 8935
Telemetry [candidate] (8.878 ms) : 0, 8878
IAST [baseline] (23.206 ms) : 0, 23206
IAST [candidate] (23.178 ms) : 0, 23178
section profiling
ProfilingAgent [baseline] (101.274 ms) : 0, 101274
ProfilingAgent [candidate] (102.227 ms) : 0, 102227
BytebuddyAgent [baseline] (710.721 ms) : 0, 710721
BytebuddyAgent [candidate] (710.735 ms) : 0, 710735
GlobalTracer [baseline] (350.286 ms) : 0, 350286
GlobalTracer [candidate] (351.512 ms) : 0, 351512
AppSec [baseline] (54.321 ms) : 0, 54321
AppSec [candidate] (53.558 ms) : 0, 53558
Debugger [baseline] (4.258 ms) : 0, 4258
Debugger [candidate] (4.279 ms) : 0, 4279
Remote Config [baseline] (702.995 µs) : 0, 703
Remote Config [candidate] (711.763 µs) : 0, 712
Telemetry [baseline] (8.951 ms) : 0, 8951
Telemetry [candidate] (8.893 ms) : 0, 8893
Profiling [baseline] (101.3 ms) : 0, 101300
Profiling [candidate] (102.253 ms) : 0, 102253

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-03-26T01:35:03	2025-03-26T01:42:50
git_branch	master	gary/add-llm-obs-api
git_commit_date	1742922414	1742951641
git_commit_sha	`05ccd9a`	`d16b87d`
release_version	1.48.0-SNAPSHOT~05ccd9aa10	1.48.0-SNAPSHOT~d16b87deb5
start_time	2025-03-26T01:34:49	2025-03-26T01:42:36

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1742953770	1742953770
ci_job_id	864930324	864930324
ci_pipeline_id	59986248	59986248
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-2uo-edd9-project-304-concurrent-1-tq2e76et 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-2uo-edd9-project-304-concurrent-1-tq2e76et 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 13 metrics, 17 unstable metrics.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.48.0-SNAPSHOT~d16b87deb5, baseline=1.48.0-SNAPSHOT~05ccd9aa10
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.362 ms) : 1344, 1381
.   : milestone, 1362,
appsec (1.731 ms) : 1708, 1755
.   : milestone, 1731,
appsec_no_iast (1.759 ms) : 1736, 1782
.   : milestone, 1759,
code_origins (1.689 ms) : 1662, 1716
.   : milestone, 1689,
iast (1.521 ms) : 1497, 1546
.   : milestone, 1521,
profiling (1.517 ms) : 1493, 1541
.   : milestone, 1517,
tracing (1.517 ms) : 1494, 1541
.   : milestone, 1517,
section candidate
no_agent (1.367 ms) : 1348, 1386
.   : milestone, 1367,
appsec (1.746 ms) : 1722, 1769
.   : milestone, 1746,
appsec_no_iast (1.741 ms) : 1717, 1765
.   : milestone, 1741,
code_origins (1.691 ms) : 1665, 1717
.   : milestone, 1691,
iast (1.526 ms) : 1501, 1551
.   : milestone, 1526,
profiling (1.551 ms) : 1527, 1575
.   : milestone, 1551,
tracing (1.508 ms) : 1482, 1533
.   : milestone, 1508,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.362 ms [1.344 ms, 1.381 ms]	-
appsec	1.731 ms [1.708 ms, 1.755 ms]	369.152 µs (27.1%)
appsec_no_iast	1.759 ms [1.736 ms, 1.782 ms]	396.408 µs (29.1%)
code_origins	1.689 ms [1.662 ms, 1.716 ms]	326.552 µs (24.0%)
iast	1.521 ms [1.497 ms, 1.546 ms]	158.976 µs (11.7%)
profiling	1.517 ms [1.493 ms, 1.541 ms]	154.781 µs (11.4%)
tracing	1.517 ms [1.494 ms, 1.541 ms]	154.794 µs (11.4%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.367 ms [1.348 ms, 1.386 ms]	-
appsec	1.746 ms [1.722 ms, 1.769 ms]	379.047 µs (27.7%)
appsec_no_iast	1.741 ms [1.717 ms, 1.765 ms]	374.401 µs (27.4%)
code_origins	1.691 ms [1.665 ms, 1.717 ms]	324.112 µs (23.7%)
iast	1.526 ms [1.501 ms, 1.551 ms]	158.962 µs (11.6%)
profiling	1.551 ms [1.527 ms, 1.575 ms]	183.976 µs (13.5%)
tracing	1.508 ms [1.482 ms, 1.533 ms]	140.667 µs (10.3%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.48.0-SNAPSHOT~d16b87deb5, baseline=1.48.0-SNAPSHOT~05ccd9aa10
    dateFormat X
    axisFormat %s
section baseline
no_agent (383.872 µs) : 364, 404
.   : milestone, 384,
iast (515.738 µs) : 494, 538
.   : milestone, 516,
iast_FULL (733.98 µs) : 712, 756
.   : milestone, 734,
iast_GLOBAL (568.696 µs) : 546, 591
.   : milestone, 569,
iast_HARDCODED_SECRET_DISABLED (519.428 µs) : 497, 542
.   : milestone, 519,
iast_INACTIVE (469.275 µs) : 448, 491
.   : milestone, 469,
iast_TELEMETRY_OFF (499.795 µs) : 478, 522
.   : milestone, 500,
tracing (463.721 µs) : 443, 485
.   : milestone, 464,
section candidate
no_agent (384.388 µs) : 363, 406
.   : milestone, 384,
iast (517.752 µs) : 496, 540
.   : milestone, 518,
iast_FULL (732.548 µs) : 711, 755
.   : milestone, 733,
iast_GLOBAL (558.019 µs) : 536, 580
.   : milestone, 558,
iast_HARDCODED_SECRET_DISABLED (518.475 µs) : 497, 540
.   : milestone, 518,
iast_INACTIVE (469.984 µs) : 448, 492
.   : milestone, 470,
iast_TELEMETRY_OFF (505.841 µs) : 484, 528
.   : milestone, 506,
tracing (466.307 µs) : 445, 487
.   : milestone, 466,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	383.872 µs [364.137 µs, 403.607 µs]	-
iast	515.738 µs [493.803 µs, 537.674 µs]	131.866 µs (34.4%)
iast_FULL	733.98 µs [712.072 µs, 755.887 µs]	350.107 µs (91.2%)
iast_GLOBAL	568.696 µs [546.035 µs, 591.357 µs]	184.824 µs (48.1%)
iast_HARDCODED_SECRET_DISABLED	519.428 µs [497.258 µs, 541.598 µs]	135.556 µs (35.3%)
iast_INACTIVE	469.275 µs [447.707 µs, 490.844 µs]	85.403 µs (22.2%)
iast_TELEMETRY_OFF	499.795 µs [477.959 µs, 521.631 µs]	115.923 µs (30.2%)
tracing	463.721 µs [442.638 µs, 484.803 µs]	79.849 µs (20.8%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	384.388 µs [362.779 µs, 405.996 µs]	-
iast	517.752 µs [495.845 µs, 539.659 µs]	133.364 µs (34.7%)
iast_FULL	732.548 µs [710.582 µs, 754.514 µs]	348.16 µs (90.6%)
iast_GLOBAL	558.019 µs [536.346 µs, 579.693 µs]	173.631 µs (45.2%)
iast_HARDCODED_SECRET_DISABLED	518.475 µs [496.903 µs, 540.047 µs]	134.087 µs (34.9%)
iast_INACTIVE	469.984 µs [447.934 µs, 492.035 µs]	85.596 µs (22.3%)
iast_TELEMETRY_OFF	505.841 µs [483.91 µs, 527.773 µs]	121.454 µs (31.6%)
tracing	466.307 µs [445.259 µs, 487.355 µs]	81.919 µs (21.3%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	gary/add-llm-obs-api
git_commit_date	1742922414	1742951641
git_commit_sha	`05ccd9a`	`d16b87d`
release_version	1.48.0-SNAPSHOT~05ccd9aa10	1.48.0-SNAPSHOT~d16b87deb5

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1742954260	1742954260
ci_job_id	864930325	864930325
ci_pipeline_id	59986248	59986248
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-2uo-edd9-project-304-concurrent-2-aezaow5s 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-2uo-edd9-project-304-concurrent-2-aezaow5s 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.48.0-SNAPSHOT~d16b87deb5, baseline=1.48.0-SNAPSHOT~05ccd9aa10
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.477 ms) : 1466, 1488
.   : milestone, 1477,
appsec (2.353 ms) : 2310, 2397
.   : milestone, 2353,
iast (2.127 ms) : 2072, 2183
.   : milestone, 2127,
iast_GLOBAL (2.163 ms) : 2108, 2219
.   : milestone, 2163,
profiling (1.98 ms) : 1937, 2024
.   : milestone, 1980,
tracing (1.951 ms) : 1909, 1994
.   : milestone, 1951,
section candidate
no_agent (1.477 ms) : 1465, 1488
.   : milestone, 1477,
appsec (2.342 ms) : 2299, 2386
.   : milestone, 2342,
iast (2.125 ms) : 2070, 2180
.   : milestone, 2125,
iast_GLOBAL (2.175 ms) : 2119, 2231
.   : milestone, 2175,
profiling (2.003 ms) : 1957, 2048
.   : milestone, 2003,
tracing (1.962 ms) : 1919, 2004
.   : milestone, 1962,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.477 ms [1.466 ms, 1.488 ms]	-
appsec	2.353 ms [2.31 ms, 2.397 ms]	876.334 µs (59.3%)
iast	2.127 ms [2.072 ms, 2.183 ms]	650.256 µs (44.0%)
iast_GLOBAL	2.163 ms [2.108 ms, 2.219 ms]	686.185 µs (46.5%)
profiling	1.98 ms [1.937 ms, 2.024 ms]	503.41 µs (34.1%)
tracing	1.951 ms [1.909 ms, 1.994 ms]	474.112 µs (32.1%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.477 ms [1.465 ms, 1.488 ms]	-
appsec	2.342 ms [2.299 ms, 2.386 ms]	865.237 µs (58.6%)
iast	2.125 ms [2.07 ms, 2.18 ms]	648.238 µs (43.9%)
iast_GLOBAL	2.175 ms [2.119 ms, 2.231 ms]	698.247 µs (47.3%)
profiling	2.003 ms [1.957 ms, 2.048 ms]	525.727 µs (35.6%)
tracing	1.962 ms [1.919 ms, 2.004 ms]	484.79 µs (32.8%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.48.0-SNAPSHOT~d16b87deb5, baseline=1.48.0-SNAPSHOT~05ccd9aa10
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.276 s) : 15276000, 15276000
.   : milestone, 15276000,
appsec (14.99 s) : 14990000, 14990000
.   : milestone, 14990000,
iast (18.897 s) : 18897000, 18897000
.   : milestone, 18897000,
iast_GLOBAL (17.951 s) : 17951000, 17951000
.   : milestone, 17951000,
profiling (15.089 s) : 15089000, 15089000
.   : milestone, 15089000,
tracing (14.99 s) : 14990000, 14990000
.   : milestone, 14990000,
section candidate
no_agent (15.019 s) : 15019000, 15019000
.   : milestone, 15019000,
appsec (14.819 s) : 14819000, 14819000
.   : milestone, 14819000,
iast (19.292 s) : 19292000, 19292000
.   : milestone, 19292000,
iast_GLOBAL (17.889 s) : 17889000, 17889000
.   : milestone, 17889000,
profiling (15.306 s) : 15306000, 15306000
.   : milestone, 15306000,
tracing (15.051 s) : 15051000, 15051000
.   : milestone, 15051000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.276 s [15.276 s, 15.276 s]	-
appsec	14.99 s [14.99 s, 14.99 s]	-286.0 ms (-1.9%)
iast	18.897 s [18.897 s, 18.897 s]	3.621 s (23.7%)
iast_GLOBAL	17.951 s [17.951 s, 17.951 s]	2.675 s (17.5%)
profiling	15.089 s [15.089 s, 15.089 s]	-187.0 ms (-1.2%)
tracing	14.99 s [14.99 s, 14.99 s]	-286.0 ms (-1.9%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.019 s [15.019 s, 15.019 s]	-
appsec	14.819 s [14.819 s, 14.819 s]	-200.0 ms (-1.3%)
iast	19.292 s [19.292 s, 19.292 s]	4.273 s (28.5%)
iast_GLOBAL	17.889 s [17.889 s, 17.889 s]	2.87 s (19.1%)
profiling	15.306 s [15.306 s, 15.306 s]	287.0 ms (1.9%)
tracing	15.051 s [15.051 s, 15.051 s]	32.0 ms (0.2%)

mcculls · 2025-01-02T20:52:04Z

dd-trace-api/src/main/java/datadog/trace/api/llmobs/LLMObs.java

+  private static LLMObsSpanFactory SPAN_FACTORY = NoOpLLMObsSpanFactory.INSTANCE;
+
+  /**
+   * This a hook for injecting SpanFactory implementation. It should only be used internally by


FYI, internal APIs should be added to the internal-api module.

You can keep the same package structure and they will still be visible to the tracer internals.

ah gotcha, thanks!

github-actions · 2025-01-14T13:24:28Z

Hi! 👋 Looks like you updated a Git Submodule.
If this was not intentional please make sure to:

Update the submodule to the latest commit on the master branch using the git submodule update command,
Check you properly set up your environment for contributing.

github-actions · 2025-01-14T13:27:35Z

Hi! 👋 Looks like you updated a Git Submodule.
If this was not intentional please make sure to:

Update the submodule to the latest commit on the master branch using the git submodule update command,
Check you properly set up your environment for contributing.

gary-huang force-pushed the gary/add-llm-obs-api branch from 84eb184 to 85d0946 Compare December 30, 2024 16:41

gary-huang force-pushed the gary/add-llm-obs-api branch from 85d0946 to dd637fd Compare January 2, 2025 19:08

mcculls reviewed Jan 2, 2025

View reviewed changes

mcculls force-pushed the gary/add-llmobs-configs branch from 14f71b3 to 5d90450 Compare January 7, 2025 09:35

Base automatically changed from gary/add-llmobs-configs to master January 7, 2025 15:36

gary-huang force-pushed the gary/add-llm-obs-api branch from dd637fd to 85debfc Compare January 14, 2025 13:24

gary-huang force-pushed the gary/add-llm-obs-api branch from 85debfc to 98060a9 Compare January 14, 2025 13:27

gary-huang force-pushed the gary/add-llm-obs-api branch from 98060a9 to 62bc9a0 Compare January 14, 2025 13:28

gary-huang force-pushed the gary/add-llm-obs-api branch from 2b23b61 to 08b677e Compare January 30, 2025 21:43

gary-huang force-pushed the gary/add-llm-obs-api branch from 08b677e to 5b12d55 Compare February 12, 2025 06:48

gary-huang force-pushed the gary/add-llm-obs-api branch from 5b12d55 to 77ddec8 Compare February 20, 2025 21:15

gary-huang force-pushed the gary/add-llm-obs-api branch 6 times, most recently from c1f63dc to 6e250a2 Compare March 7, 2025 13:34

gary-huang added 2 commits March 25, 2025 21:14

add APIs for llm obs

e2f347c

add llm message class to support llm spans

d16b87d

gary-huang force-pushed the gary/add-llm-obs-api branch from 6e250a2 to d16b87d Compare March 26, 2025 01:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add APIs for llm obs #8135

add APIs for llm obs #8135

gary-huang commented Dec 30, 2024 •

edited by Kyle-Verhoog

Loading

pr-commenter bot commented Dec 30, 2024 •

edited

Loading

mcculls Jan 2, 2025

gary-huang Jan 3, 2025

github-actions bot commented Jan 14, 2025

github-actions bot commented Jan 14, 2025

add APIs for llm obs #8135

Are you sure you want to change the base?

add APIs for llm obs #8135

Conversation

gary-huang commented Dec 30, 2024 • edited by Kyle-Verhoog Loading

What Does This Do

Motivation

Additional Notes

Contributor Checklist

pr-commenter bot commented Dec 30, 2024 • edited Loading

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

mcculls Jan 2, 2025

Choose a reason for hiding this comment

gary-huang Jan 3, 2025

Choose a reason for hiding this comment

github-actions bot commented Jan 14, 2025

github-actions bot commented Jan 14, 2025

gary-huang commented Dec 30, 2024 •

edited by Kyle-Verhoog

Loading

pr-commenter bot commented Dec 30, 2024 •

edited

Loading