Skip to content

Conversation

@sergeytimoshin
Copy link
Contributor

@sergeytimoshin sergeytimoshin commented Oct 29, 2025

Summary by CodeRabbit

  • New Features

    • Photon gRPC client and WorkCoordinator for real-time queue updates; CLI/config options to set Photon gRPC URL and gRPC port (default 50051)
  • Improvements

    • Event-driven processing with polling fallback and coordinator-based tree registration
    • More robust batch pagination, stricter validation for address/proof ordering, and widened queue/index ranges
  • Observability

    • Expanded debug/logging and telemetry across queue, slot, and proof processing
  • Tests / Chores

    • Protobuf build integration, test updates, and dev-dependency additions for gRPC paths

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Oct 29, 2025

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

📥 Commits

Reviewing files that changed from the base of the PR and between 15dcf48 and 94ca5ca.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (41)
  • .github/actions/setup-and-build/action.yml (1 hunks)
  • cli/src/commands/test-validator/index.ts (2 hunks)
  • cli/src/utils/initTestEnv.ts (3 hunks)
  • cli/src/utils/processPhotonIndexer.ts (2 hunks)
  • forester-utils/Cargo.toml (1 hunks)
  • forester-utils/src/instructions/address_batch_update.rs (10 hunks)
  • forester-utils/src/lib.rs (1 hunks)
  • forester/Cargo.toml (2 hunks)
  • forester/build.rs (1 hunks)
  • forester/proto/photon.proto (1 hunks)
  • forester/src/cli.rs (1 hunks)
  • forester/src/config.rs (4 hunks)
  • forester/src/epoch_manager.rs (21 hunks)
  • forester/src/forester_status.rs (2 hunks)
  • forester/src/lib.rs (2 hunks)
  • forester/src/processor/v1/helpers.rs (1 hunks)
  • forester/src/processor/v1/send_transaction.rs (1 hunks)
  • forester/src/processor/v2/common.rs (2 hunks)
  • forester/src/slot_tracker.rs (1 hunks)
  • forester/src/tree_data_sync.rs (2 hunks)
  • forester/src/work_coordinator.rs (1 hunks)
  • forester/tests/e2e_test.rs (3 hunks)
  • forester/tests/legacy/batched_address_test.rs (1 hunks)
  • forester/tests/legacy/batched_state_async_indexer_test.rs (1 hunks)
  • forester/tests/legacy/batched_state_indexer_test.rs (1 hunks)
  • forester/tests/legacy/batched_state_test.rs (1 hunks)
  • forester/tests/legacy/e2e_test.rs (2 hunks)
  • forester/tests/legacy/e2e_v1_test.rs (2 hunks)
  • forester/tests/priority_fee_test.rs (1 hunks)
  • forester/tests/test_batch_append_spent.rs (1 hunks)
  • forester/tests/test_utils.rs (1 hunks)
  • program-tests/compressed-token-test/tests/v1.rs (1 hunks)
  • program-tests/system-cpi-v2-test/tests/event.rs (1 hunks)
  • prover/client/src/proof_types/batch_address_append/proof_inputs.rs (1 hunks)
  • scripts/devenv/versions.sh (1 hunks)
  • sdk-libs/client/src/indexer/photon_indexer.rs (1 hunks)
  • sdk-libs/client/src/lib.rs (1 hunks)
  • sdk-libs/client/src/local_test_validator.rs (3 hunks)
  • sdk-libs/photon-api/src/models/_get_queue_elements_post_200_response_result.rs (1 hunks)
  • sdk-libs/photon-api/src/models/account_context.rs (1 hunks)
  • sdk-tests/client-test/tests/light_client.rs (1 hunks)
 ______________________________________________________________________________________________________________________
< Adapting old programs to fit new machines usually means adapting new machines to behave like old ones. - Alan Perlis >
 ----------------------------------------------------------------------------------------------------------------------
  \
   \   \
        \ /\
        ( )
      .( o ).

Walkthrough

This PR integrates Photon gRPC service connectivity into Forester for event-driven queue updates. A new WorkCoordinator component connects to Photon, subscribes to queue updates, and distributes them to registered trees via per-tree channels. Configuration propagates the Photon gRPC URL through CLI and config layers. Proto definitions define the QueueService API. Supporting changes include v2 batch processing refactoring with parallel proof generation, addition of batch_start_index fields, and type conversions for queue indices.

Changes

Cohort / File(s) Summary
Photon gRPC Infrastructure
forester/proto/photon.proto, forester/build.rs
New Proto definition for QueueService with GetQueueInfo and SubscribeQueueUpdates RPCs. Adds messages for queue info and subscription control. Build script configures tonic-prost to compile proto definitions.
WorkCoordinator Module
forester/src/work_coordinator.rs
New async gRPC coordination component subscribing to Photon queue updates and routing them to registered trees via per-tree channels. Includes dispatcher loop, health tracking, reconnection logic with exponential backoff, and tree registration/unregistration lifecycle.
Dependency Updates
forester/Cargo.toml, forester-utils/Cargo.toml
Added tonic, prost, prost-types, tonic-prost, tokio-stream to support gRPC. Added build-dependency tonic-prost-build. Added dev-dependencies tokio-postgres and bs58 to forester-utils.
Configuration & CLI
forester/src/cli.rs, forester/src/config.rs
Added photon_grpc_url field to StartArgs and ExternalServicesConfig, sourced from FORESTER_PHOTON_GRPC_URL environment variable. RpcPoolConfig now derives Default.
EpochManager Coordinator Integration
forester/src/epoch_manager.rs
Added coordinator field to EpochManager for event-driven processing. Wired initialization when photon_grpc_url configured. Added process_queue_v2_event path alongside polling fallback. Introduced should_skip_tree helper and tree lifecycle registration/unregistration. Added test scaffolding for configuration helpers.
V2 Batch Processing Refactor
forester-utils/src/instructions/address_batch_update.rs, forester/src/processor/v2/common.rs, forester-utils/src/lib.rs
Replaced sequential proof batching with parallel proof generation using join_all. Introduced next_queue_index pagination. Added batch_start_index field to ParsedMerkleTreeData. Enhanced debug logging for observability.
Queue Index Type Updates
sdk-libs/photon-api/src/models/_get_queue_elements_post_200_response_result.rs, sdk-libs/photon-api/src/models/account_context.rs, sdk-libs/client/src/indexer/photon_indexer.rs
Changed first_value_queue_index and nullifier_queue_index types from u16 to u64. Updated corresponding assignments and constructors.
Status & Tracking Updates
forester/src/forester_status.rs, forester/src/slot_tracker.rs
Filtered rolled-over trees from fullness reporting and forester assignment checks. Direct actual slot reads in wait_until_slot_reached with MAX_SLEEP_SLOTS cap and adaptive sleep duration.
Tree Data Sync & Validation
forester/src/tree_data_sync.rs
Added has_discriminator helper to validate BatchedMerkleTreeAccount discriminators before deserialization, gating state/address extraction.
Debug & Logging
forester/src/processor/v1/helpers.rs, forester/src/processor/v1/send_transaction.rs
Added debug-only state tree deserialization block in fetch_proofs_and_create_instructions (disabled by default). Elevated transaction send failure log level from warn to error.
CLI & Indexer Setup
cli/src/utils/processPhotonIndexer.ts
Added --grpc-port 50051 argument to indexer startup command.
Test Configuration
forester/tests/e2e_test.rs, forester/tests/priority_fee_test.rs, forester/tests/test_utils.rs
Updated test configs to initialize photon_grpc_url field. Added get_photon_grpc_url helper for test mode selection (Local vs. Devnet).
Batch Address Validation
prover/client/src/proof_types/batch_address_append/proof_inputs.rs
Added ordering validation loop in get_batch_address_append_circuit_inputs to enforce low < new < high across batch elements with error on violation.

Sequence Diagram(s)

sequenceDiagram
    participant EM as EpochManager
    participant WC as WorkCoordinator
    participant Photon as Photon gRPC
    participant Tree as Tree Task
    
    rect rgb(200, 220, 255)
    Note over EM,Photon: Initialization (photon_grpc_url present)
    EM->>WC: new(photon_grpc_url)
    WC->>Photon: Establish gRPC connection
    Photon-->>WC: Channel ready
    EM->>WC: run_dispatcher()
    WC->>WC: spawn async dispatcher loop
    end
    
    rect rgb(200, 220, 255)
    Note over WC,Tree: Event-driven Processing
    WC->>Photon: SubscribeQueueUpdates
    Photon-->>WC: stream QueueUpdate
    WC->>WC: register_tree(tree_pubkey)
    WC-->>Tree: mpsc::Receiver<QueueUpdateMessage>
    Photon-->>WC: Queue update event
    WC->>Tree: Send update via channel
    Tree->>Tree: Process event-driven work
    end
    
    rect rgb(255, 220, 200)
    Note over WC,Photon: Fallback on Disconnect
    WC->>Photon: Reconnect attempt (exponential backoff)
    alt Reconnect succeeds
        Photon-->>WC: New connection
    else Max retries exceeded
        WC->>WC: Fall back to polling
        Note over WC: EpochManager switches to polling mode
    end
    end
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~30 minutes

  • forester/src/work_coordinator.rs — New async gRPC component with dispatcher loop, reconnection logic, and channel-based routing; requires careful review of concurrency patterns and error handling
  • forester-utils/src/instructions/address_batch_update.rs — Significant refactoring from sequential to parallel proof generation with join_all; parallelization semantics and pagination logic need verification
  • forester/src/epoch_manager.rs — Event-driven vs. fallback processing paths; coordinator initialization and tree registration/unregistration lifecycle; test coverage for should_skip_tree
  • Type conversions (u16 → u64) — Verify consistency across all queue index usages (photon_indexer.rs, account_context.rs, batch response types)
  • Proto compilation & build.rs — Ensure proto file path resolution and generated code integration are correct

Possibly related PRs

Suggested reviewers

  • SwenSchaeferjohann

Poem

🐰 A coordinator hops through Photon's streams,
Gathering queue updates in parallel beams,
Batches now fly faster, proofs aligned,
Event-driven work, no polling assigned,
Forester's fleet now dances with gRPC's grace!

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 53.73% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check ✅ Passed The PR title "feat: forester: grpc processing for v2 trees" directly corresponds to the primary objective of this changeset. The modifications introduce comprehensive gRPC-based event-driven processing infrastructure for V2 trees, including a new WorkCoordinator module, protobuf definitions for a Photon QueueService, configuration support for photon_grpc_url, and integration throughout the epoch manager and related processing logic. While the PR includes supporting changes such as parallel proof generation, type refinements, and observability improvements, these are all foundational to enabling the core feature described in the title. The title is concise, specific, and clearly communicates the main change without vague terminology.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@sergeytimoshin sergeytimoshin changed the title feat(forester): add grpc processing for v2 trees feat: forester: add grpc processing for v2 trees Oct 29, 2025
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 12

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
cli/src/utils/processPhotonIndexer.ts (1)

55-62: Fix incorrect gRPC flag and value type; use --grpc-url with full URL endpoint

The code uses wrong flag name and value type. Photon CLI uses --grpc-url (not --grpc-port) to configure gRPC endpoint. The current "--grpc-port", "50051" passes a bare port instead of a URL, causing the flag to be unrecognized or ignored.

Required changes:

  • Flag: --grpc-port--grpc-url
  • Value: "50051" → full URL like "http://localhost:50051"
  • Add optional grpcUrl parameter to function signature

Suggested fix:

+export const PHOTON_GRPC_URL = "http://localhost:50051";

export async function startIndexer(
  rpcUrl: string,
  indexerPort: number,
  checkPhotonVersion: boolean = true,
  photonDatabaseUrl?: string,
+  grpcUrl: string = PHOTON_GRPC_URL,
) {
   ...
   const args: string[] = [
     "--port",
     indexerPort.toString(),
     "--rpc-url",
     rpcUrl,
-    "--grpc-port",
-    "50051",
+    "--grpc-url",
+    grpcUrl,
   ];
🧹 Nitpick comments (15)
forester/src/processor/v1/helpers.rs (1)

172-260: Consider removing or making the debug code configurable.

This debug block is currently dead code since _debug is hardcoded to false. While the diagnostic logic for validating on-chain state and root history could be valuable for troubleshooting, it adds ~90 lines of unmaintained code with no activation mechanism.

Consider one of the following:

  1. Remove the debug block if it's no longer needed for active development.
  2. Make it configurable via an environment variable or feature flag so it can be enabled without source modifications:
-        let _debug = false;
-        if _debug {
+        if std::env::var("FORESTER_DEBUG_STATE_PROOFS").is_ok() {

Additionally, this PR focuses on v2 tree processing according to the title, but these changes are in the v1 helpers. Please clarify if this debug code is needed for v2 work or if it was left in accidentally.

forester-utils/Cargo.toml (1)

51-53: Dev deps look fine; consider constraining features/pinning via workspace

To keep compile surface minimal and versions consistent:

  • Prefer default-features = false on tokio-postgres unless TLS is needed via companion crates.
  • Pin bs58 in the workspace and reference a version here (already using workspace = true).

Example:

[dev-dependencies]
tokio-postgres = { version = "0.7", default-features = false }
bs58 = { workspace = true }
forester/src/forester_status.rs (2)

198-203: Use centralized skip logic (should_skip_tree) for consistency

Prefer the new helper to encapsulate skip rules and avoid drift if criteria expand.

For example:

-        // Skip rolled-over trees
-        if tree.is_rolledover {
+        // Skip trees not eligible for processing
+        if crate::epoch_manager::should_skip_tree(tree) {
             continue;
         }

Confirm run_queue_info earlier doesn’t need the same filter to keep metrics and status aligned.


256-259: Build active_trees via helper to avoid duplicating criteria

Construct the filtered list using should_skip_tree to ensure the same rules apply everywhere.

-    // Filter out rolled-over trees
-    let active_trees: Vec<TreeAccounts> =
-        trees.iter().filter(|t| !t.is_rolledover).cloned().collect();
+    // Filter out ineligible trees
+    let active_trees: Vec<TreeAccounts> =
+        trees.iter().filter(|t| !crate::epoch_manager::should_skip_tree(t)).cloned().collect();
forester/src/slot_tracker.rs (1)

86-113: Harden against transient RPC errors and consider parameterizing MAX_SLEEP_SLOTS

The loop currently bails on any get_slot() error; brief RPC blips will error out the caller.

  • Wrap get_slot() in a small retry/backoff and continue on error.
  • Make MAX_SLEEP_SLOTS configurable (env/arg) so tests and different slot times can tune it.
  • Optional: compute sleep without floats:
-    const MAX_SLEEP_SLOTS: u64 = 50; // ~20 seconds max sleep between checks
+    const MAX_SLEEP_SLOTS: u64 = 50; // ~20–23s depending on slot duration

@@
-        let sleep_duration =
-            Duration::from_secs_f64(sleep_slots as f64 * slot_duration().as_secs_f64());
+        let per_slot = slot_duration();
+        let sleep_duration = per_slot.saturating_mul(sleep_slots as u32);

Where saturating_mul is available from Duration on recent Rust; otherwise multiply nanos with checked math.

If keeping errors fatal is intentional (e.g., upstream retries), please confirm.

forester-utils/src/instructions/address_batch_update.rs (3)

146-159: Bound parallel proof generation; preserve output order

join_all fires all requests at once; large chunks can overwhelm the prover and memory.

  • Limit concurrency with a buffer (e.g., PROVER_MAX_CONCURRENCY).
  • Preserve order by indexing results and emitting in-order.

Illustrative sketch:

-            let proof_futures: Vec<_> = all_inputs.into_iter().enumerate().map(|(i, inputs)| {
-                let client = Arc::clone(&proof_client);
-                async move {
-                    let result = client.generate_batch_address_append_proof(inputs).await;
-                    (i, result)
-                }
-            }).collect();
-
-            // Wait for all proofs to complete in parallel
-            let proof_results = futures::future::join_all(proof_futures).await;
+            let max_conc = std::cmp::max(1, std::env::var("PROVER_MAX_CONCURRENCY").ok()
+                .and_then(|v| v.parse::<usize>().ok()).unwrap_or(8));
+            let proof_results = futures::stream::iter(
+                all_inputs.into_iter().enumerate().map(|(i, inputs)| {
+                    let client = Arc::clone(&proof_client);
+                    async move { (i, client.generate_batch_address_append_proof(inputs).await) }
+                })
+            )
+            .buffer_unordered(max_conc)
+            .collect::<Vec<_>>()
+            .await;
+            let mut proof_results = {
+                let mut v = proof_results;
+                v.sort_by_key(|(i, _)| *i);
+                v
+            };

This keeps memory bounded and preserves the intended proof order.


175-179: Avoid cloning proof_buffer when yielding

Clone allocates and copies every time; use mem::take to move out efficiently.

-                        if proof_buffer.len() >= MAX_PROOFS_PER_TX {
-                            yield Ok(proof_buffer.clone());
-                            proof_buffer.clear();
-                        }
+                        if proof_buffer.len() >= MAX_PROOFS_PER_TX {
+                            let out = std::mem::take(&mut proof_buffer);
+                            yield Ok(out);
+                        }

74-87: Minor: clarify param naming (queue index vs offset)

Trait signature uses start_offset but we pass a queue index; name the local var start_queue_index (or add a comment) to avoid confusion across indexer backends.

-            let indexer_update_info = {
+            let indexer_update_info = {
                 let mut connection = rpc_pool.get_connection().await?;
                 let indexer = connection.indexer_mut()?;
                 debug!(
                     "Requesting {} addresses from Photon for chunk {} with start_queue_index={:?}",
                     elements_for_chunk, chunk_idx, next_queue_index
                 );
                 match indexer
                 .get_address_queue_with_proofs(
                     &merkle_tree_pubkey,
                     elements_for_chunk as u16,
                     next_queue_index,
                     None,
                 )
forester/src/processor/v2/common.rs (2)

522-533: Reduce debug payload and avoid leaking large arrays in logs

Current debug logs dump entire hash chains; this is noisy and can bloat logs in active phases.

Consider logging only count + short prefixes/suffixes, or guard with a higher verbosity flag:

  • e.g., map each hash chain to hex_prefix(8) and cap to first/last N entries.
  • Or move detailed dumps to trace!.

516-521: Avoid potential panic on direct indexing of hash_chain_stores

Accessing hash_chain_stores[batch_index as usize][i as usize] can panic if on-chain data is inconsistent.

Apply safe access with get() and skip missing entries:

-        for i in num_inserted_zkps..current_zkp_batch_index {
-            leaves_hash_chains
-                .push(merkle_tree.hash_chain_stores[batch_index as usize][i as usize]);
-        }
+        if let Some(batch_store) = merkle_tree.hash_chain_stores.get(batch_index as usize) {
+            for i in num_inserted_zkps..current_zkp_batch_index {
+                if let Some(h) = batch_store.get(i as usize) {
+                    leaves_hash_chains.push(*h);
+                } else {
+                    trace!("Missing hash chain entry at i={} (batch_index={})", i, batch_index);
+                }
+            }
+        } else {
+            trace!("Missing hash_chain_stores row for batch_index={}", batch_index);
+        }
forester/Cargo.toml (2)

56-62: Enable TLS for gRPC (if using https endpoints)

If photon_grpc_url can be https://…, enable tonic TLS features or connections will fail at runtime.

Apply:

-tonic = "0.14.2"
+tonic = { version = "0.14.2", features = ["tls", "tls-roots"] }

If Photon is strictly plaintext (http), ignore this.


56-65: Question: Do we need the tonic-prost runtime crate?

Code only uses tonic client types; generation happens in build.rs via tonic-prost-build. The tonic-prost runtime dep appears unused.

Consider removing tonic-prost from [dependencies] to avoid extra surface area unless it’s used elsewhere.

forester/src/epoch_manager.rs (2)

944-967: Redundant use_events check

use_events equals queue_update_rx.is_some(), then checked again inside the V2 branch.

Inline queue_update_rx.is_some() and drop use_events for clarity.


1350-1487: Event loop is solid; ensure resilience when coordinator restarts

If the coordinator restarts and drops senders, recv() returns None. Your select! won’t match that branch; fallback tick covers this. Consider logging once if recv() returns None to aid debugging.

forester/src/work_coordinator.rs (1)

117-126: Optional: subscribe per-iteration with a fresh client

If you prefer to avoid interior mutability, re-create the client per loop iteration:

let mut stream = QueueServiceClient::connect(self.photon_grpc_url.clone())
    .await?
    .subscribe_queue_updates(request)
    .await?
    .into_inner();

This removes the need for reconnect() and the grpc_client field.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between e7bdf0f and 15dcf48.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (25)
  • cli/src/utils/processPhotonIndexer.ts (1 hunks)
  • forester-utils/Cargo.toml (1 hunks)
  • forester-utils/src/instructions/address_batch_update.rs (10 hunks)
  • forester-utils/src/lib.rs (1 hunks)
  • forester/Cargo.toml (2 hunks)
  • forester/build.rs (1 hunks)
  • forester/proto/photon.proto (1 hunks)
  • forester/src/cli.rs (1 hunks)
  • forester/src/config.rs (4 hunks)
  • forester/src/epoch_manager.rs (20 hunks)
  • forester/src/forester_status.rs (2 hunks)
  • forester/src/lib.rs (2 hunks)
  • forester/src/processor/v1/helpers.rs (1 hunks)
  • forester/src/processor/v1/send_transaction.rs (1 hunks)
  • forester/src/processor/v2/common.rs (2 hunks)
  • forester/src/slot_tracker.rs (1 hunks)
  • forester/src/tree_data_sync.rs (4 hunks)
  • forester/src/work_coordinator.rs (1 hunks)
  • forester/tests/e2e_test.rs (2 hunks)
  • forester/tests/priority_fee_test.rs (1 hunks)
  • forester/tests/test_utils.rs (1 hunks)
  • prover/client/src/proof_types/batch_address_append/proof_inputs.rs (1 hunks)
  • sdk-libs/client/src/indexer/photon_indexer.rs (1 hunks)
  • sdk-libs/photon-api/src/models/_get_queue_elements_post_200_response_result.rs (1 hunks)
  • sdk-libs/photon-api/src/models/account_context.rs (1 hunks)
🧰 Additional context used
🧠 Learnings (8)
📚 Learning: 2025-10-11T21:59:25.222Z
Learnt from: CR
PR: Lightprotocol/light-protocol#0
File: program-libs/account-checks/CLAUDE.md:0-0
Timestamp: 2025-10-11T21:59:25.222Z
Learning: Applies to program-libs/account-checks/src/**/*.rs : Validate account type with 8-byte discriminators using check_discriminator before deserialization

Applied to files:

  • forester/src/tree_data_sync.rs
📚 Learning: 2025-10-11T21:59:25.222Z
Learnt from: CR
PR: Lightprotocol/light-protocol#0
File: program-libs/account-checks/CLAUDE.md:0-0
Timestamp: 2025-10-11T21:59:25.222Z
Learning: Applies to program-libs/account-checks/src/discriminator.rs : Define and keep discriminator constants and the Discriminator trait in discriminator.rs for compile-time verification

Applied to files:

  • forester/src/tree_data_sync.rs
📚 Learning: 2025-10-11T21:59:52.712Z
Learnt from: CR
PR: Lightprotocol/light-protocol#0
File: program-libs/account-checks/docs/CLAUDE.md:0-0
Timestamp: 2025-10-11T21:59:52.712Z
Learning: Applies to program-libs/account-checks/docs/**/DISCRIMINATOR.md : DISCRIMINATOR.md must document 8-byte discriminators, the Discriminator trait, constant arrays for compile-time checks, and integration with account initialization

Applied to files:

  • forester/src/tree_data_sync.rs
📚 Learning: 2025-10-11T21:59:25.222Z
Learnt from: CR
PR: Lightprotocol/light-protocol#0
File: program-libs/account-checks/CLAUDE.md:0-0
Timestamp: 2025-10-11T21:59:25.222Z
Learning: Applies to program-libs/account-checks/src/**/*.rs : On account initialization, call account_info_init to set the 8-byte discriminator

Applied to files:

  • forester/src/tree_data_sync.rs
📚 Learning: 2025-10-11T21:59:25.222Z
Learnt from: CR
PR: Lightprotocol/light-protocol#0
File: program-libs/account-checks/CLAUDE.md:0-0
Timestamp: 2025-10-11T21:59:25.222Z
Learning: Applies to program-libs/account-checks/src/checks.rs : Expose and maintain account validation helpers (check_owner, check_program, check_mut/non_mut, check_signer, check_discriminator, set_discriminator, check_pda_seeds, check_account_balance_is_rent_exempt, account_info_init) in checks.rs

Applied to files:

  • forester/src/tree_data_sync.rs
📚 Learning: 2025-10-15T03:46:26.767Z
Learnt from: CR
PR: Lightprotocol/light-protocol#0
File: programs/registry/CLAUDE.md:0-0
Timestamp: 2025-10-15T03:46:26.767Z
Learning: Applies to programs/registry/src/lib.rs : Load accounts according to type before check_forester: batched via BatchedMerkleTreeAccount::type_from_account_info(); regular via ctx.accounts.account.load()?.metadata; use custom deserialization when required.

Applied to files:

  • forester/src/tree_data_sync.rs
📚 Learning: 2025-10-16T06:33:19.426Z
Learnt from: CR
PR: Lightprotocol/light-protocol#0
File: program-libs/compressible/CLAUDE.md:0-0
Timestamp: 2025-10-16T06:33:19.426Z
Learning: Applies to program-libs/compressible/src/config.rs : Ensure serialization compatibility across Anchor, Pinocchio, and Borsh for core account types used by dependent programs

Applied to files:

  • forester/src/tree_data_sync.rs
📚 Learning: 2025-10-11T21:59:25.222Z
Learnt from: CR
PR: Lightprotocol/light-protocol#0
File: program-libs/account-checks/CLAUDE.md:0-0
Timestamp: 2025-10-11T21:59:25.222Z
Learning: Applies to program-libs/account-checks/**/Cargo.toml : Define features solana, pinocchio, and test-only in Cargo.toml; default build should enable none

Applied to files:

  • forester/Cargo.toml
🧬 Code graph analysis (7)
forester/src/processor/v1/helpers.rs (2)
program-tests/merkle-tree/src/lib.rs (1)
  • root (190-195)
program-libs/zero-copy/src/cyclic_vec.rs (1)
  • first_index (200-206)
prover/client/src/proof_types/batch_address_append/proof_inputs.rs (1)
prover/client/tests/batch_address_append.rs (1)
  • new_element_values (64-67)
forester-utils/src/instructions/address_batch_update.rs (7)
sdk-libs/client/src/indexer/photon_indexer.rs (2)
  • get_address_queue_with_proofs (1473-1579)
  • new (116-127)
sdk-libs/program-test/src/indexer/test_indexer.rs (3)
  • get_address_queue_with_proofs (875-959)
  • clone (113-125)
  • new (1309-1370)
sdk-libs/client/src/indexer/indexer_trait.rs (1)
  • get_address_queue_with_proofs (179-185)
sdk-libs/client/src/indexer/types.rs (3)
  • value (373-377)
  • value (379-383)
  • value (830-841)
sdk-libs/client/src/indexer/error.rs (1)
  • clone (113-159)
prover/client/src/proof_client.rs (1)
  • generate_batch_address_append_proof (525-533)
program-libs/hasher/src/hash_chain.rs (1)
  • create_hash_chain_from_slice (23-32)
forester/src/processor/v2/common.rs (1)
sdk-libs/program-test/src/logging/config.rs (1)
  • debug (59-71)
forester/src/epoch_manager.rs (5)
forester/src/work_coordinator.rs (2)
  • new (45-60)
  • is_healthy (192-194)
forester/src/slot_tracker.rs (3)
  • new (25-35)
  • wait_until_slot_reached (79-116)
  • update (37-44)
sdk-libs/client/src/rpc/rpc_trait.rs (2)
  • new (34-42)
  • new (76-78)
forester-utils/src/forester_epoch.rs (3)
  • new (68-80)
  • new (156-161)
  • get_epoch_phases (232-273)
forester/src/config.rs (7)
  • clone (353-368)
  • default (89-100)
  • default (142-149)
  • default (153-158)
  • default (162-172)
  • test_address_v2 (104-115)
  • test_state_v2 (117-128)
forester/src/tree_data_sync.rs (2)
program-libs/batched-merkle-tree/src/queue.rs (1)
  • light_account_checks (184-184)
program-libs/batched-merkle-tree/src/merkle_tree.rs (4)
  • light_account_checks (137-137)
  • BatchedMerkleTreeAccount (1080-1080)
  • BatchedMerkleTreeAccount (1090-1090)
  • BatchedMerkleTreeAccount (1100-1100)
forester/src/work_coordinator.rs (1)
forester/src/epoch_manager.rs (2)
  • new (135-195)
  • e (573-573)
🪛 Buf (1.59.0)
forester/proto/photon.proto

3-3: Files with package "photon" must be within a directory "photon" relative to root but were in directory "forester/proto".

(PACKAGE_DIRECTORY_MATCH)

🔇 Additional comments (21)
sdk-libs/photon-api/src/models/account_context.rs (1)

27-27: LGTM: Queue index widened to u64.

The type change from u16 to u64 for nullifier_queue_index aligns with the v2 API migration and accommodates larger queue indices.

sdk-libs/client/src/indexer/photon_indexer.rs (1)

1658-1658: LGTM: Cast removed following API type change.

The direct assignment is now correct since api_result.first_value_queue_index is u64, matching the expected type.

sdk-libs/photon-api/src/models/_get_queue_elements_post_200_response_result.rs (1)

18-18: LGTM: Queue index widened to u64.

Both the field type and constructor parameter are consistently updated from u16 to u64, supporting larger queue indices in the v2 API.

Also applies to: 26-26

forester/src/processor/v1/send_transaction.rs (1)

354-354: LGTM: Improved logging consistency.

Elevating the log level to error! for transaction send failures when a signature is present improves consistency with line 356 and appropriately reflects the severity when a submitted transaction fails.

forester/src/cli.rs (1)

71-72: LGTM!

The photon_grpc_url field is correctly added following the same pattern as other optional URL fields in the struct.

forester/src/lib.rs (2)

22-22: LGTM!

The new work_coordinator module is correctly exposed as a public module, enabling event-driven queue updates via gRPC.


66-66: LGTM!

Adding !t.is_rolledover to the filter correctly excludes rolled-over trees from queue processing, preventing unnecessary work on inactive trees.

forester/tests/test_utils.rs (1)

92-92: LGTM!

Setting photon_grpc_url: None in the test configuration is appropriate for tests that don't require gRPC connectivity.

forester/tests/e2e_test.rs (2)

124-129: LGTM!

The get_photon_grpc_url() helper follows the established pattern of other URL helpers, correctly returning the local gRPC endpoint for local testing and reading from environment for Devnet.


212-212: LGTM!

The photon_grpc_url field is correctly populated from the helper function, maintaining consistency with the configuration pattern.

forester/tests/priority_fee_test.rs (1)

55-55: LGTM!

Setting photon_grpc_url: None is appropriate for this priority fee test that doesn't require gRPC connectivity.

forester/src/config.rs (4)

43-43: LGTM!

The photon_grpc_url field is correctly added to ExternalServicesConfig, maintaining consistency with other optional service URLs.


236-236: LGTM!

The photon_grpc_url is correctly propagated from StartArgs into ExternalServicesConfig during configuration initialization.


315-315: LGTM!

Setting photon_grpc_url: None in new_for_status is correct, as status checks don't require gRPC connectivity.


131-131: The Default derive concern is technically valid but not a current runtime issue.

The derived Default on RpcPoolConfig (line 131) would indeed produce all-zero values if called, which differs from the intentional hardcoded values (max_size: 10, connection_timeout_secs: 15, etc.). However, verification found:

  • Both instantiation points (lines 280 and 336) use explicit full field initialization
  • No Default::default() calls on RpcPoolConfig exist anywhere in the codebase
  • No struct update syntax or implicit patterns could trigger Default
  • RpcPoolConfig is only used within config.rs

The derived Default is currently unused dead code. While the concern is valid and removing Default or implementing it properly would be defensive programming, this doesn't pose an immediate runtime risk with current usage patterns.

forester/build.rs (1)

1-6: No issues found. Proto file verified to exist and build script is correctly implemented.

The verification confirms that forester/proto/photon.proto exists at the expected location with valid proto3 syntax. The build script correctly references this file with the proper paths ("proto/photon.proto" and include directory "proto"). The implementation appropriately handles dependencies with the println! directive for cargo:rerun-if-changed.

forester-utils/src/lib.rs (1)

25-25: No action required—batch_start_index initialization is verified at all construction sites.

The verification confirms there is exactly one construction site for ParsedMerkleTreeData in the codebase (forester/src/processor/v2/common.rs:534–544), and it properly initializes the new batch_start_index field from batch.start_index at line 542. No Default implementation exists that could bypass initialization, and all other references to the struct are type annotations or imports, not construction sites.

forester/src/processor/v2/common.rs (1)

542-542: LGTM: batch_start_index populated

Populating batch_start_index from batch.start_index closes data-gap for downstream consumers.

forester/src/epoch_manager.rs (2)

146-179: Startup coordinator wiring looks good; keep running even if initial connect fails

Graceful fallback to polling with clear logs. No changes requested.


1993-2001: Nice: centralized skip logic with tests

Helper + tests improve maintainability and scheduling clarity.

forester/proto/photon.proto (1)

3-3: Verify and apply Buf package-directory fix

The Buf linting rule PACKAGE_DIRECTORY_MATCH is correctly identified: package photon at forester/proto/photon.proto violates the convention requiring package names to match directory structure.

Recommended option: Move file to forester/proto/photon/photon.proto, keeping package photon; and the existing include_proto!("photon") call (it resolves by package name, not path). This requires no Rust code changes.

Alternative: Change package declaration to photon_proto or similar to match the current path, then update the include_proto!() call in forester/src/work_coordinator.rs:20 accordingly.

Verify your project's proto organization conventions and Buf enforcement before proceeding.

@sergeytimoshin sergeytimoshin changed the title feat: forester: add grpc processing for v2 trees feat: forester: grpc processing for v2 trees Oct 30, 2025
Copy link
Contributor

@SwenSchaeferjohann SwenSchaeferjohann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants