Skip to content

feat: queue time capture for Rack#2838

Merged
sl0thentr0py merged 26 commits intomasterfrom
puma-queue-time
Feb 25, 2026
Merged

feat: queue time capture for Rack#2838
sl0thentr0py merged 26 commits intomasterfrom
puma-queue-time

Conversation

@dingsdax
Copy link
Contributor

@dingsdax dingsdax commented Jan 15, 2026

Captures how long requests wait for a e.g. Puma thread from X-Request-Start headers and attaches it to transactions as http.server.request.time_in_queue. Sentry only measures after Puma picks up the request. Under load, requests might wait long in the queue but only take a fraction to process. Similar to what judoscale and scout do.

works with all major reverse proxies:

  • Nginx: X-Request-Start: t=1234567890.123 (seconds)
  • Heroku: X-Request-Start: t=1234567890123456 (microseconds)
  • HAProxy 1.9+: X-Request-Start: t=1234567890123456 (microseconds)
  • HAProxy < 1.9: X-Request-Start: t=1234567890 (seconds)
  • Generic: Raw timestamps in seconds/milliseconds/microseconds

subtracts puma.request_body_wait for accuracy (excludes slow client uploads from queue time).

Usage

works 🤞 if your reverse proxy sets the header:

# Nginx
proxy_set_header X-Request-Start "t=${msec}";

opt-out

config.capture_queue_time = false
sentry being sentry

@linear
Copy link

linear bot commented Jan 15, 2026

@github-actions
Copy link

github-actions bot commented Jan 15, 2026

Messages
📖 Do not forget to update Sentry-docs with your feature once the pull request gets approved.

Generated by 🚫 dangerJS against 3cc6f8a

@dingsdax dingsdax requested review from sl0thentr0py and solnic and removed request for solnic January 15, 2026 10:22
@dingsdax dingsdax marked this pull request as ready for review January 15, 2026 10:23
MESSAGING_MESSAGE_RETRY_COUNT = "messaging.message.retry.count"

# Time in ms the request spent in the server queue before processing began.
HTTP_QUEUE_TIME_MS = "http.queue_time_ms"
Copy link
Member

@sl0thentr0py sl0thentr0py Jan 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AbhiPrasad @lcian is there an attribute convention for time spent in queue waiting for the http request to be picked up in an HTTP server?

The closest I could find is http.client.request.time_in_queue but that's for clients.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm I cannot find anything like that. Maybe we introduce our own http.server.request.time_in_queue counterpart?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sounds good; shall I add to sentry-conventions too?

Copy link
Member

@sl0thentr0py sl0thentr0py left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some small stuff, and mainly want to wait for feedback on the attribute convention name

Copy link
Collaborator

@solnic solnic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ but I left a bunch of minor 💅🏻 suggestions 🙃

dingsdax and others added 4 commits January 26, 2026 16:25
Co-authored-by: Peter Solnica <peter@solnica.online>
Co-authored-by: Peter Solnica <peter@solnica.online>
Non-numeric t= values like "t=invalid" or "t=" were silently converted
to 0.0 by Ruby's String#to_f, resulting in a Unix epoch timestamp and a
queue time of ~56 years instead of nil.

Apply the same numeric regex guard already used for raw timestamps to
the t= branch before calling to_f.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-authored-by: Peter Solnica <peter@solnica.online>
dingsdax and others added 5 commits February 24, 2026 14:33
Co-authored-by: Peter Solnica <peter@solnica.online>
Co-authored-by: Peter Solnica <peter@solnica.online>
Co-authored-by: Peter Solnica <peter@solnica.online>
Co-authored-by: Peter Solnica <peter@solnica.online>
Co-authored-by: Peter Solnica <peter@solnica.online>
dingsdax and others added 5 commits February 24, 2026 14:35
parse_request_start_header returned nil for headers that arrived with
leading/trailing whitespace or as comma-separated values (multiple
header occurrences collapsed by a proxy). Strip whitespace from the
token and split on commas, taking the first (earliest) timestamp.

Co-Authored-By: Claude <noreply@anthropic.com>
When env["puma.request_body_wait"] is a String (e.g. serialized by
middleware), the `> 0` comparison raised ArgumentError, which was
silently swallowed by the broad rescue, causing queue time to be
dropped even with a valid X-Request-Start header.

Call to_f on String values before the comparison. Numeric strings
("40") convert correctly; non-numeric strings ("N/A") become 0.0
and are treated as absent, so total_time_ms is returned unchanged.

Co-Authored-By: Claude <noreply@anthropic.com>
Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.

queue_time_ms >= 0 ? queue_time_ms : 0.0 # more sanity check
else
total_time_ms
end
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Queue time spoofable via client header

Medium Severity

extract_queue_time trusts env["HTTP_X_REQUEST_START"] without verifying it was set/overwritten by a trusted proxy and without any upper-bound sanity check, so a client-supplied X-Request-Start can inject arbitrarily large http.server.request.time_in_queue values and skew transaction data.

Fix in Cursor Fix in Web

dingsdax added a commit to getsentry/sentry-conventions that referenced this pull request Feb 24, 2026
Adds the `http.server.request.time_in_queue` semantic convention attribute
to capture the time (in milliseconds) a request spent waiting in the server
queue before processing began.

This attribute is populated from the `X-Request-Start` header set by reverse
proxies (Nginx, HAProxy, Heroku router, etc.) and first introduced in
sentry-ruby: getsentry/sentry-ruby#2838

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@sl0thentr0py sl0thentr0py merged commit 6525d0b into master Feb 25, 2026
260 of 261 checks passed
@sl0thentr0py sl0thentr0py deleted the puma-queue-time branch February 25, 2026 11:33
dingsdax added a commit to getsentry/sentry-conventions that referenced this pull request Feb 25, 2026
* feat(http): add http.server.request.time_in_queue attribute

Adds the `http.server.request.time_in_queue` semantic convention attribute
to capture the time (in milliseconds) a request spent waiting in the server
queue before processing began.

This attribute is populated from the `X-Request-Start` header set by reverse
proxies (Nginx, HAProxy, Heroku router, etc.) and first introduced in
sentry-ruby: getsentry/sentry-ruby#2838

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(http): add generated attribute definitions for http.server.request.time_in_queue

Generate TypeScript and Python attribute definitions including type
exports, metadata entries, and TypedDict annotations.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Puma request queue time as a transaction attribute

4 participants