Make notification and cross chain queue sizes grow with network size

[ndr-ds]

Motivation

Proposal

Test Plan

Release Plan

• Nothing to do / These changes follow the usual release cycle.
• These changes should be backported to the latest devnet branch, then
  • be released in a new SDK,
  • be released in a validator hotfix.
• These changes should be backported to the latest testnet branch, then
  • be released in a new SDK,
  • be released in a validator hotfix.
• Backporting is not possible but we may want to deploy a new devnet and release a new
  SDK soon.

Links

• reviewer checklist

[ndr-ds]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• Send notifications in batches #5054
• Make notification and cross chain queue sizes grow with network size #5053 👈 (View in Graphite)
• Adding notification metrics #5052
• Make tempo an actual chart dependency #5051
• Add node local DNS cache #5050
• Add Otel collector to our tracing exports to Tempo #5049
• Improve dashboards #5048
• Check block cache on create_cross_chain_requests #5013
• Measure operation/message execution latency in microseconds #5007
• Add cross chain message tasks to dashboard #4999
• Add create network actions latency to dashboard #5008 : 1 other dependent PR (#5010 )
• Add create network actions latency metric #5006
• Add cross chain queue wait time and chain worker request queue wait time #5000
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.