Skip to content

docs: add disabling monitoring guide and benchmark report #2244

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cowsking wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs: add disabling monitoring guide and benchmark report #2244

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions docs/custom-metric-filter.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -307,9 +307,77 @@ curl http://localhost:8675/metrics
- **Config Sync v1.11.x and earlier**: Uses `stackdriver` exporter (deprecated)
- **Config Sync v1.8+**: Custom monitoring support available

## Disabling Monitoring Telemetry

Starting in Config Sync v1.25, users can opt out of metric collection to save resource consumption (CPU/memory) and reduce telemetry overheads.

### Option 1: Per-Sync Declarative Disabling (RootSync / RepoSync)

You can turn off monitoring for individual `RootSync` or `RepoSync` objects by setting `spec.monitoring.enabled: false` in their specification:

> **Note**: The default `RootSync` object named `root-sync` created by Google Cloud management tools (like the Google Cloud console, Google Cloud CLI, or Terraform) does not support the `spec.monitoring.enabled` field. Any changes to the default `root-sync` object will be automatically reverted by the management controller. You can set `spec.monitoring.enabled: false` on custom `RootSync` objects (with names other than `root-sync`) or any `RepoSync` object.

```yaml
apiVersion: configsync.gke.io/v1beta1
kind: RootSync # or RepoSync
metadata:
name: SYNC_NAME
namespace: SYNC_NAMESPACE
spec:
monitoring:
enabled: false
```

When `spec.monitoring.enabled: false` is set:
- The `otel-agent` sidecar container is omitted from the reconciler Pod, saving container memory and CPU requests/limits.
- Metric collection and exporting for that reconciler are turned off.

### Option 2: Complete Cluster-Wide Removal (OSS Administrators)

If an open-source cluster administrator wants to safely delete the `config-management-monitoring` namespace entirely and stop all OpenTelemetry controller activity:

1. Set the `DISABLE_MONITORING=true` environment variable on both `reconciler-manager` and `resource-group-controller-manager` Deployments using `kubectl set env` (or by updating their Deployment manifests):

```shell
kubectl set env deployment/reconciler-manager -n config-management-system DISABLE_MONITORING=true
kubectl set env deployment/resource-group-controller-manager -n resource-group-system DISABLE_MONITORING=true
```

Alternatively, inject the environment variable directly into the Deployment manifests:

```yaml
env:
- name: DISABLE_MONITORING
Comment thread
tiffanny29631 marked this conversation as resolved.
value: "true"
```

2. Delete the monitoring namespace:
Comment thread
tiffanny29631 marked this conversation as resolved.

> **Note**: Before deleting the `config-management-monitoring` namespace, verify that the `DISABLE_MONITORING=true` rollout has completed on the manager controllers (so that active reconciler Pods no longer contain the `otel-agent` container) to ensure no reconcilers attempt to export metrics to the deleted monitoring namespace.

```shell
kubectl delete ns config-management-monitoring
```

### Benchmark & Resource Savings (500 RepoSync Scale)

The following benchmark measures the memory savings from disabling the OpenTelemetry (`otel-agent`) sidecar on a cluster with 500 `RepoSync` objects across 500 namespaces on an 8-node GKE cluster:

| Metric / Scope | Monitoring Enabled (Baseline) | Monitoring Disabled (Optimized) | Total Memory Saved | Average Savings per RepoSync |
|----------------|-------------------------------|----------------------------------|--------------------|------------------------------|
| **Total Memory (`config-management-system`)** | 42,187 Mi (~41.20 GiB) | 25,542 Mi (~24.94 GiB) | **16,645 Mi (~16.25 GiB)** | **~33.29 MiB (-39.5%)** |
| **Containers per Reconciler Pod** | 3/3 (`reconciler`, `git-sync`, `otel-agent`) | 2/2 (`reconciler`, `git-sync`) | **-1 Sidecar Container** | **-100% Sidecar Overhead** |
| **Telemetry Network Overhead** | Active gRPC to `:4317` | None (Noop) | **Zero Exporter Overhead** | **100% Telemetry Offloaded** |

#### Benchmark Methodology & Highlights
- **Environment**: 8-node GKE cluster running 500 active `RepoSync` instances across 500 namespaces (`test-ns-1` through `test-ns-500`).
- **Measurement**: Raw aggregated Pod memory footprint measured via `kubectl top pods -n config-management-system`.
- **Key Finding**: Disabling the OpenTelemetry sidecar across 500 `RepoSync` reconcilers reclaims **16,645 Mi (~16.25 GiB)** of memory back to the cluster (saving **~33.29 MiB** per reconciler Pod).

## Additional Resources

- [OpenTelemetry Collector Documentation](https://opentelemetry.io/docs/collector/)
- [Google Cloud Monitoring Documentation](https://cloud.google.com/monitoring)
- [Config Sync Monitoring Guide](http://cloud/anthos-config-management/docs/how-to/monitoring-config-sync)
- [Available Config Sync Metrics](http://cloud/anthos-config-management/docs/how-to/monitoring-config-sync#metrics)