DOC-11497 Docs for obs: Enabling troubleshooting hot spots externally (e.g., logs or metrics) #19577
Conversation
Files changed:
|
✅ Deploy Preview for cockroachdb-interactivetutorials-docs canceled.
|
✅ Deploy Preview for cockroachdb-api-docs canceled.
|
❌ Deploy Preview for cockroachdb-docs failed. Why did it fail? →
|
✅ Netlify Preview
To edit notification comments on pull requests, go to your Netlify project configuration. |
There was a problem hiding this comment.
thanks for the first look, @angles-n-daemons please review again.
There was a problem hiding this comment.
Few questions and suggestions,
- Can we simplify this and remove the second box ("Is there a node outlier in the metrics?")?
- Are guaranteed to have a 'hot ranges log' when there is a popular key log for the latch contention workflow? CC @angles-n-daemons
There was a problem hiding this comment.
modified diagram
There was a problem hiding this comment.
We aren't, I'll explain in detail why.
The hot ranges log shows up under two conditions when enabled:
- The logging interval duration has elapsed (eg, once every four hours).
- A single replica has exceeded the CPU threshold we configured for logging.
Now when there's a popular key, or rather a row hotspot, a single range may be receiving most of the traffic, but much of the incoming queries are waiting for a latch to be released rather than doing anything. Waiting for a latch incurs no effect on cpu utilization, so if there are lots of waiting queries, there's not quite as much cpu activity.
You can see this difference in the Anatomy of a Hotspot document, if you look at "Appendix B: Anatomy of a Row Hotspot", you'll see that while elevated, the cpu utilization for the leaseholder doesn't exceed 25%.
It's certainly possible that this is enough to go over the threshold defined, but not guaranteed.
There was a problem hiding this comment.
@kevin-v-ngo thanks for your first review, please take a second look.
There was a problem hiding this comment.
modified diagram
Fixes DOC-11497
Added detect-hotspots.md and associated images.
Rendered previews: