feat(autoscaler): Add an optional in-memory source for the autoscaler's active-function list.

[dmikhaylovnv]

TL;DR

Add an optional in-memory source for the autoscaler's active-function list. This lets each node discover functions from the metrics backend and scale only functions in its assigned Cassandra buckets. The self-managed stack enables this mode by default when HELMFILE_ENV=stg; other environments continue to use Cassandra.

Additional Details

The scaling loop currently reads active functions from Cassandra. Local discovery provides a staging path for evaluating per-node discovery while retaining the existing Cassandra discovery writer, membership, locks, and positive-capacity TTL writes.

• Add scaling.active_function_source with cassandra as the application default.
• Add a per-node registry that retains positive observations, expires entries after the configured retention period, and drops entries when bucket ownership changes.
• Match Cassandra's partition-key hashing when assigning discovered function versions to buckets.
• Wait for initial local discovery before scaling in local mode, and recheck bucket ownership before processing requests.
• Add registry and discovery metrics. Log and fail the process if the local discovery task exits unexpectedly.
• Reject a zero discovery interval at startup.
• Document the mode and its configuration. Allow an explicit environment override, including cassandra in staging.

Local mode adds metrics-backend discovery queries on each autoscaler node. Its registry is rebuilt after a restart.

For the Reviewer

Please focus on the Cassandra-compatible bucket calculation, registry retention across partial discovery results, and ownership checks around scaling requests.

Customer Release Notes

Staging self-managed deployments use per-node function discovery by default. Other environments retain the Cassandra active-function read path unless explicitly configured otherwise.

Plan Summary

No new Kubernetes resources. Staging changes the autoscaler environment configuration and may increase metrics-backend query load with each replica.

Usage

Set functionAutoscaler.env.SCALING__ACTIVE_FUNCTION_SOURCE to local_discovery or cassandra to override the stack default. A pod restart applies the change.

Summary by CodeRabbit

• New Features
  • Function autoscaling can use either Cassandra or per-node local discovery to identify active functions. Cassandra remains the default, while self-managed staging environments default to local discovery; other environments continue to use Cassandra.
  • Local discovery can be selected or overridden through configuration. It tracks recently active functions assigned to each node and waits for an initial successful discovery before using the local registry.
• Bug Fixes
  • Invalid zero-length discovery intervals are rejected at startup rather than allowing autoscaler startup with an unusable interval.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: NVIDIA/nvcf/.coderabbit.yaml
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 9e236c57-df11-419a-bda9-49e2a696ecea

📥 Commits

Reviewing files that changed from the base of the PR and between e77241c and 785309a.

📒 Files selected for processing: 11

• deploy/stacks/self-managed/global.yaml.gotmpl
• deploy/stacks/self-managed/tests/observability-autoscaler.sh
• src/control-plane-services/function-autoscaler/LOCAL_DISCOVERY.md
• src/control-plane-services/function-autoscaler/crates/server/resources/settings-local.yaml
• src/control-plane-services/function-autoscaler/crates/server/src/metrics/mod.rs
• src/control-plane-services/function-autoscaler/crates/server/src/scaling/mod.rs
• src/control-plane-services/function-autoscaler/crates/server/src/server.rs
• src/control-plane-services/function-autoscaler/crates/server/src/work/bucket.rs
• src/control-plane-services/function-autoscaler/crates/server/src/work/discovery.rs
• src/control-plane-services/function-autoscaler/crates/server/src/work/local_registry.rs
• src/control-plane-services/function-autoscaler/crates/server/src/work/mod.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

📝 Walkthrough

The function autoscaler can select Cassandra or local discovery as its active-function source. In local mode, it periodically discovers functions, stores bucket-scoped results in an expiring registry, and supplies those results to autoscaling. The self-managed stage stack defaults to local discovery.

Changes: Local function discovery

Layer / File(s)	Summary
Source selection and deployment defaults src/control-plane-services/function-autoscaler/crates/server/src/scaling/mod.rs, src/control-plane-services/function-autoscaler/crates/server/resources/settings-local.yaml, deploy/stacks/self-managed/global.yaml.gotmpl, deploy/stacks/self-managed/tests/observability-autoscaler.sh, src/control-plane-services/function-autoscaler/LOCAL_DISCOVERY.md	ScalingSettings adds a cassandra-defaulted source setting and validates the discovery interval. The self-managed stage stack defaults to local_discovery; tests check that default and the cassandra override. Documentation describes the modes and registry behavior.
Bucket mapping and local registry src/control-plane-services/function-autoscaler/crates/server/src/work/bucket.rs, src/control-plane-services/function-autoscaler/crates/server/src/work/discovery.rs, src/control-plane-services/function-autoscaler/crates/server/src/work/local_registry.rs, src/control-plane-services/function-autoscaler/crates/server/src/metrics/mod.rs	Bucket helpers map function/version UUIDs to Cassandra buckets and check node ownership. LocalFunctionRegistry filters discoveries by ownership, retains unexpired entries, and exposes bucket results. New metrics track registry size, discovery counts, and failures. Tests cover mapping, ownership, expiry, and retention.
Discovery task and autoscaling integration src/control-plane-services/function-autoscaler/crates/server/src/server.rs, src/control-plane-services/function-autoscaler/crates/server/src/work/mod.rs	In local mode, the server starts periodic discovery and passes the registry to P0 autoscaling. Autoscaling waits for registry initialization, selects functions from the configured source, and checks bucket ownership during processing. The server reports discovery-task termination as an error.

Priority: ⬇️ Low

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant AutoscalerServer
  participant DiscoveryTask
  participant TimeseriesDB
  participant LocalFunctionRegistry
  participant P0Autoscaling
  AutoscalerServer->>DiscoveryTask: Start periodic discovery
  DiscoveryTask->>TimeseriesDB: Fetch recently invoked functions
  TimeseriesDB-->>DiscoveryTask: Return active function details
  DiscoveryTask->>LocalFunctionRegistry: Merge results for owned buckets
  P0Autoscaling->>LocalFunctionRegistry: Read functions for a bucket
  LocalFunctionRegistry-->>P0Autoscaling: Return active functions

Loading

Merge Risk: ⚪ Minimal · up to 78530

The supported deployment workflow preserves the configured source, and scaling waits for successful initial discovery. No merge-blocking issue is established.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 43.90% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 41 functions across 8 files. (3 skipped: …	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title uses the required Conventional Commits format with one type prefix, a scope, and a subject. The feat type accurately describes the primary feature: an optional in-memory source for the autos…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Title check

Explanation

The title uses the required Conventional Commits format with one type prefix, a scope, and a subject. The feat type accurately describes the primary feature: an optional in-memory source for the autoscaler’s active-function list.

Full details: Docstring Coverage

Explanation

Docstring coverage is 43.90% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 41 functions across 8 files. (3 skipped: 3 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

• Fix all pre-merge checks with AI

✨ Finishing Touches: 💡 1

📝 Generate docstrings: 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🛡️ CodeQL Analysis

✅ No security issues found!

🔗 View full details in Security tab

_{🕐 Last updated: 2026-10-16 21:43:41 UTC | Commit: 785309a}