Skip to content

feat: initialize metrics on startup#1812

Open
Thegaram wants to merge 1 commit into
developfrom
feat-initialize-metrics-to-avoid-alerts
Open

feat: initialize metrics on startup#1812
Thegaram wants to merge 1 commit into
developfrom
feat-initialize-metrics-to-avoid-alerts

Conversation

@Thegaram

@Thegaram Thegaram commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Several Prometheus block-height gauges (batch/chunk propose height, L2 relayer commit height, L1/L2 watcher processed height) reset to 0 on restart, so lag alerts fired until the next event repopulated them. This seeds each gauge from the DB at startup so it reflects the real height immediately.

An orthogonal fix is to add a > 0 condition to the alert rules. But initializing the metrics here seems more correct.

Summary by CodeRabbit

  • New Features

    • Updated the application version to v4.7.16.
    • Improved monitoring accuracy by restoring key processing and proposal metrics after service restarts.
    • Metrics now reflect the latest persisted progress for batches, chunks, and L1/L2 block processing.
  • Bug Fixes

    • Prevented startup metrics from incorrectly resetting or showing missing progress after recovery.
    • Improved handling of unavailable or missing historical data during metric initialization.

@Thegaram Thegaram requested review from georgehao and johnsonjie July 15, 2026 14:54
@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Constructors now seed rollup Prometheus gauges from persisted database or resumed watcher state. A committed-batch lookup supports relayer initialization, and the internal version tag advances to v4.7.16.

Changes

Startup metric initialization

Layer / File(s) Summary
Persisted batch metric source
rollup/internal/orm/batch.go, rollup/internal/controller/watcher/batch_proposer.go
Adds a latest-committed-batch query and uses it to initialize the batch proposer block-height gauge.
Watcher metric seeding
rollup/internal/controller/watcher/chunk_proposer.go, rollup/internal/controller/watcher/l1_watcher.go, rollup/internal/controller/watcher/l2_watcher.go
Seeds chunk, L1 processed-block, and L2 missing-block gauges during watcher construction.
Relayer metric seeding
rollup/internal/controller/relayer/l2_relayer.go
Initializes the relayer committed-block gauge from the latest committed batch’s end chunk.

Version update

Layer / File(s) Summary
Version tag update
common/version/version.go
Updates the internal version tag from v4.7.15 to v4.7.16.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

Suggested labels: bump-version

Suggested reviewers: georgehao, johnsonjie

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description explains the goal and approach, but it omits the template's required sections for PR title, deployment tag versioning, and breaking change. Add the missing template sections with short answers for purpose/design rationale, PR title, deployment tag versioning, and breaking change status.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title is conventional and accurately summarizes the main change: initializing metrics on startup.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat-initialize-metrics-to-avoid-alerts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
rollup/internal/controller/watcher/chunk_proposer.go (1)

135-148: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Ignore gorm.ErrRecordNotFound to prevent startup warnings on a fresh database.

Like BatchProposer.initializeMetrics, consider explicitly ignoring gorm.ErrRecordNotFound. Without this check, starting the service with an empty database will trigger a warning log since GetLatestChunk returns ErrRecordNotFound when no chunks exist yet.

♻️ Proposed refactor
 func (p *ChunkProposer) initializeMetrics(ctx context.Context) {
 	latestChunk, err := p.chunkOrm.GetLatestChunk(ctx)
 	if err != nil {
+		if errors.Is(err, gorm.ErrRecordNotFound) {
+			return // no chunks proposed yet, nothing to seed
+		}
 		log.Warn("failed to initialize chunk propose block height metric", "err", err)
 		return
 	}

Ensure the "errors" and "gorm.io/gorm" packages are imported at the top of the file if you apply this fix.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rollup/internal/controller/watcher/chunk_proposer.go` around lines 135 - 148,
Update ChunkProposer.initializeMetrics to treat gorm.ErrRecordNotFound from
GetLatestChunk as an expected empty-database condition: detect it with errors.Is
and return without logging, while preserving the warning for other errors and
the metric update for an existing latestChunk. Add the required errors and gorm
imports.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@rollup/internal/controller/watcher/chunk_proposer.go`:
- Around line 135-148: Update ChunkProposer.initializeMetrics to treat
gorm.ErrRecordNotFound from GetLatestChunk as an expected empty-database
condition: detect it with errors.Is and return without logging, while preserving
the warning for other errors and the metric update for an existing latestChunk.
Add the required errors and gorm imports.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 2cda3dd4-02e1-4b88-9350-e9276d4756f9

📥 Commits

Reviewing files that changed from the base of the PR and between 79c1f8c and dee9965.

📒 Files selected for processing (7)
  • common/version/version.go
  • rollup/internal/controller/relayer/l2_relayer.go
  • rollup/internal/controller/watcher/batch_proposer.go
  • rollup/internal/controller/watcher/chunk_proposer.go
  • rollup/internal/controller/watcher/l1_watcher.go
  • rollup/internal/controller/watcher/l2_watcher.go
  • rollup/internal/orm/batch.go

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 44.00000% with 42 lines in your changes missing coverage. Please review.
✅ Project coverage is 35.44%. Comparing base (79c1f8c) to head (dee9965).

Files with missing lines Patch % Lines
rollup/internal/orm/batch.go 0.00% 13 Missing ⚠️
rollup/internal/controller/relayer/l2_relayer.go 33.33% 11 Missing and 1 partial ⚠️
...llup/internal/controller/watcher/batch_proposer.go 44.44% 8 Missing and 2 partials ⚠️
...llup/internal/controller/watcher/chunk_proposer.go 63.63% 3 Missing and 1 partial ⚠️
rollup/internal/controller/watcher/l2_watcher.go 66.66% 1 Missing and 2 partials ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1812      +/-   ##
===========================================
+ Coverage    35.42%   35.44%   +0.02%     
===========================================
  Files          262      262              
  Lines        22523    22596      +73     
===========================================
+ Hits          7979     8010      +31     
- Misses       13712    13748      +36     
- Partials       832      838       +6     
Flag Coverage Δ
rollup 35.18% <44.00%> (+0.07%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants