Add new metrics for litellm by steveny91 · Pull Request #24955 · DataDog/integrations-core

steveny91 · 2026-08-22T19:07:57Z

What does this PR do?

Add new metrics for litellm tested with version 1.9.5

datadog-datadog-us1-prod · 2026-08-22T19:09:52Z

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 97.24% (+8.58%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 51ada47 | Docs | View more details | Give us feedback!}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 55bdd12d4e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

cit-pr-commenter-54b7da · 2026-08-22T19:17:58Z

evalya-impact-summary

evalya impact analysis

Impact analysis: 0 selected, 0 skipped (of 0 test tasks)
Publish tasks:   1 (always emitted)
Diff (6 files):
  litellm/changelog.d/24955.added
  litellm/datadog_checks/litellm/metrics.py
  litellm/metadata.csv
  litellm/tests/common.py
  litellm/tests/fixtures/metrics_v1_95.txt
  litellm/tests/test_unit.py

Debug a specific task: evalya plan impact --path <path> --task <task>

Learn more about CI impact filtering

jeff-morgan-dd

Please add metric descriptions where noted - thank you!

jeff-morgan-dd · 2026-08-24T17:56:04Z

 litellm.proxy.pre_call.latency.sum,count,,millisecond,,Latency for proxy_pre_call service,0,litellm,,,
 litellm.proxy.pre_call.total_requests.count,count,,request,,Number of requests for proxy_pre_call service in the time period,0,litellm,,,
 litellm.proxy.total_requests.metric.count,count,,request,,Number of requests made to the proxy server in the time period - track number of client side requests,0,litellm,,,
+litellm.redis.daily_agent_spend_update_queue.size,gauge,,,,litellm_redis_daily_agent_spend_update_queue_size,0,litellm,,,


Can you please update to include a brief description of what the metric does? Currently the description just repeats the metric name.

jeff-morgan-dd · 2026-08-24T17:56:09Z

 litellm.proxy.pre_call.total_requests.count,count,,request,,Number of requests for proxy_pre_call service in the time period,0,litellm,,,
 litellm.proxy.total_requests.metric.count,count,,request,,Number of requests made to the proxy server in the time period - track number of client side requests,0,litellm,,,
+litellm.redis.daily_agent_spend_update_queue.size,gauge,,,,litellm_redis_daily_agent_spend_update_queue_size,0,litellm,,,
+litellm.redis.daily_end_user_spend_update_queue.size,gauge,,,,litellm_redis_daily_end_user_spend_update_queue_size,0,litellm,,,


Can you please update to include a brief description of what the metric does? Currently the description just repeats the metric name.

Sorry forgot to push the uncommitted changes. Thanks for catching!

dd-octo-sts · 2026-08-24T18:04:40Z

Validation Report

All 21 validations passed.

Show details

Validation	Description	Status
`agent-reqs`	Verify check versions match the Agent requirements file	✅
`ci`	Validate CI configuration and code coverage settings	✅
`codeowners`	Validate every integration has a CODEOWNERS entry	✅
`config`	Validate default configuration files against spec.yaml	✅
`dep`	Verify dependency pins are consistent and Agent-compatible	✅
`http`	Validate integrations use the HTTP wrapper correctly	✅
`imports`	Validate check imports do not use deprecated modules	✅
`integration-style`	Validate check code style conventions	✅
`jmx-metrics`	Validate JMX metrics definition files and config	✅
`labeler`	Validate PR labeler config matches integration directories	✅
`legacy-signature`	Validate no integration uses the legacy Agent check signature	✅
`license-headers`	Validate Python files have proper license headers	✅
`licenses`	Validate third-party license attribution list	✅
`metadata`	Validate metadata.csv metric definitions	✅
`models`	Validate configuration data models match spec.yaml	✅
`openmetrics`	Validate OpenMetrics integrations disable the metric limit	✅
`package`	Validate Python package metadata and naming	✅
`qa-label`	Validate the pull request declares whether it needs QA for the next Agent release	✅
`readmes`	Validate README files have required sections	✅
`saved-views`	Validate saved view JSON file structure and fields	✅
`version`	Validate version consistency between package and changelog	✅

View full run

jeff-morgan-dd

Thanks for the updates!

sarah-witt · 2026-08-24T18:18:13Z

@@ -0,0 +1 @@
+Add new metrics for litellm


Is this adding support for metrics added in 1.9.5+? If so I'd make a note that we now support 1.9.5 in the changelog

sarah-witt · 2026-08-24T18:20:25Z

+
+    for metric in NEW_METRICS_V1_95:
+        aggregator.assert_metric(metric)
+


Do we have a test that calls assert_metrics_using_metadata? And could we add one if we don't?

sarah-witt · 2026-08-24T18:56:28Z

+    aggregator.assert_service_check('litellm.openmetrics.health', ServiceCheck.OK)
+
+
+def test_litellm_v1_95_new_metrics_are_mapped(dd_run_check, aggregator, mock_http_response):


what does this test verify that test_litellm_v1_95_metrics doesn't?

sarah-witt · 2026-08-24T18:57:41Z

+litellm_request_queue_time_seconds_count{api_key_alias="None",api_provider="openai",end_user="None",hashed_api_key="litellm_proxy_master_key",model="gpt-4o-cache-read",model_id="00b3700001b042d4a2e4db444b6e5b1a70197fdb0224960e846f23f576b76439",org_alias="None",org_id="None",requested_model="mock-cache-read",team="None",team_alias="None",user="default_user_id"} 3.0
+litellm_request_queue_time_seconds_sum{api_key_alias="None",api_provider="openai",end_user="None",hashed_api_key="litellm_proxy_master_key",model="gpt-4o-cache-read",model_id="00b3700001b042d4a2e4db444b6e5b1a70197fdb0224960e846f23f576b76439",org_alias="None",org_id="None",requested_model="mock-cache-read",team="None",team_alias="None",user="default_user_id"} 0.0004284381866455078
+litellm_request_queue_time_seconds_bucket{api_key_alias="None",api_provider="openai",end_user="None",hashed_api_key="litellm_proxy_master_key",le="0.005",model="gpt-4o-cache-write",model_id="b9656ec990e75de2a6df4a65bb191f7ee8afb5cd2432c1f1c880ad4ed3d2b3ce",org_alias="None",org_id="None",requested_model="mock-cache-write",team="None",team_alias="None",user="default_user_id"} 3.0
+litellm_request_queue_time_seconds_bucket{api_key_alias="None",api_provider="openai",end_user="None",hashed_api_key="litellm_proxy_master_key",le="0.01",model="gpt-4o-cache-write",model_id="b9656ec990e75de2a6df4a65bb191f7ee8afb5cd2432c1f1c880ad4ed3d2b3ce",org_alias="None",org_id="None",requested_model="mock-cache-write",team="None",team_alias="None",user="default_user_i


nit- some metrics in the payload don't have example data, is it possible to generate samples or even fake ones?

steveny91 added 2 commits August 22, 2026 14:56

add new metrics for litellm

c66bf02

changelog

55bdd12

steveny91 requested review from a team as code owners August 22, 2026 19:07

dd-octo-sts Bot added integration/litellm team/agent-integrations team/documentation labels Aug 22, 2026

chatgpt-codex-connector Bot reviewed Aug 22, 2026

View reviewed changes

Comment thread litellm/datadog_checks/litellm/metrics.py

add undocumented metrics

738d5f3

jeff-morgan-dd self-assigned this Aug 24, 2026

steveny91 added the qa/skip-qa Automatically skip this PR for the next QA label Aug 24, 2026

jeff-morgan-dd requested changes Aug 24, 2026

View reviewed changes

descriptions

51ada47

jeff-morgan-dd approved these changes Aug 24, 2026

View reviewed changes

sarah-witt reviewed Aug 24, 2026

View reviewed changes


		for metric in NEW_METRICS_V1_95:
		aggregator.assert_metric(metric)

		aggregator.assert_service_check('litellm.openmetrics.health', ServiceCheck.OK)


		def test_litellm_v1_95_new_metrics_are_mapped(dd_run_check, aggregator, mock_http_response):

Uh oh!

Conversation

steveny91 commented Aug 22, 2026

What does this PR do?

Uh oh!

datadog-datadog-us1-prod Bot commented Aug 22, 2026 • edited by datadog-prod-us1-6 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 22, 2026

evalya-impact-summary

Uh oh!

jeff-morgan-dd left a comment

Choose a reason for hiding this comment

Uh oh!

jeff-morgan-dd Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

jeff-morgan-dd Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

steveny91 Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

dd-octo-sts Bot commented Aug 24, 2026

Validation Report

Uh oh!

jeff-morgan-dd left a comment

Choose a reason for hiding this comment

Uh oh!

sarah-witt Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

sarah-witt Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

sarah-witt Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

sarah-witt Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

datadog-datadog-us1-prod Bot commented Aug 22, 2026 •

edited by datadog-prod-us1-6 Bot

Loading