docs(self-hosted): monitor self-hosted instance (#12660)

We don't have any good article or explanation on how to monitor self-hosted Sentry.

"Monitoring self-hosted Sentry" means 3 things:
1. Monitor the actual services made by Sentry (sentry, snuba, symbolicator, relay) by using Statsd
2. Monitor the dependencies (Kafka, Postgres, etc) by using Prometheus exporters, or similar things
3. Monitor the host machine or Docker daemon

## LEGAL BOILERPLATE

Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.

Author: aldy505

develop-docs/self-hosted/experimental/monitoring.mdx

@@ -0,0 +1,104 @@
+---
+title: Self-Hosted Monitoring
+sidebar_title: Monitoring
+sidebar_order: 10
+---
+
+<Alert title="Important" level="warning">
+    These are community-contributed docs. Sentry does not officially provide support for self-hosted configurations beyond the default install.
+</Alert>
+
+This page is considered experimental because everyone will have different setup and requirements for their monitoring system. It is also best to use your existing monitoring system, and try to integrate Sentry with it, instead of spinning up a new one.
+
+Most containers have a `statsd` client that you can point to your monitoring system. If you have a native `statsd` server instance, you can directly use it. If you don't, you might want to add some kind of converter that converts the ingested `statsd` format into your own. For example, if you are using Prometheus, you can use [prometheus-statsd-exporter](https://github.com/prometheus/statsd_exporter) to bridge the gap.
+
+If you are using DataDog as your monitoring system, it is recommended to use [DogStatsD](https://docs.datadoghq.com/developers/dogstatsd/?tab=hostagent) as the collector.
+
+We recommend [the original `statsd` server made by Etsy](https://github.com/statsd/statsd) or the others on the [server re-implementations](https://github.com/statsd/statsd/blob/master/docs/server_implementations.md).
+
+Sentry does not provide any alerts if your host instance is low on resources such as free memory or disk space. You will need to configure this own your own relative to your needs. It is critical for you to monitor the disk space as once it gets full, it will be much harder to recover from there.
+
+<Alert title="Note">
+    After changing configuration files, don't forget to restart the containers with `docker compose restart`. See the <Link to="/self-hosted/#configuration">configuration section</Link> for more information.
+</Alert>
+
+## Sentry-related configurations
+
+This documentation assumes `100.100.123.123` as your statsd host IP address. On most services, it is recommended to provide an IP address for the host like this, instead of a domain name like `statsd.yourcompany.com`.
+
+### Sentry
+
+You can configure Sentry to send metrics to Statsd server by configuring your `sentry/sentry.conf.py` file:
+
+```python
+SENTRY_METRICS_BACKEND = 'sentry.metrics.statsd.StatsdMetricsBackend'
+SENTRY_METRICS_OPTIONS: dict[str, Any] = {
+    'host': '100.100.123.123', # It's recommended to use IP address instead of domain name
+    'port': 8125,
+}
+# SENTRY_METRICS_SAMPLE_RATE = 1.0   # Adjust this to your needs, default is 1.0
+# SENTRY_METRICS_PREFIX = "sentry."  # Adjust this to your needs, default is "sentry."
+```
+
+### Snuba
+
+You can configure Snuba to send metrics to Statsd server by configuring your `docker-compose.yml` file:
+
+```yaml
+# The rest of your Docker Compose file
+
+x-snuba-defaults: &snuba_defaults
+  <<: *restart_policy
+  depends_on:
+    clickhouse:
+      <<: *depends_on-healthy
+    kafka:
+      <<: *depends_on-healthy
+    redis:
+      <<: *depends_on-healthy
+  image: "$SNUBA_IMAGE"
+  environment:
+    SNUBA_SETTINGS: self_hosted
+    CLICKHOUSE_HOST: clickhouse
+    DEFAULT_BROKERS: "kafka:9092"
+    REDIS_HOST: redis
+    UWSGI_MAX_REQUESTS: "10000"
+    UWSGI_DISABLE_LOGGING: "true"
+    # Leaving the value empty to just pass whatever is set
+    # on the host system (or in the .env file)
+    SENTRY_EVENT_RETENTION_DAYS:
+    SNUBA_STATSD_HOST: "100.100.123.123" # Must be an IP address, not domain name
+    SNUBA_STATSD_PORT: 8125
+
+# The rest of your Docker Compose file
+```
+
+### Relay
+
+You can configure Relay to send metrics to Statsd server by configuring your `relay/config.yml` file:
+
+```yaml
+# The rest of your config.yml file
+metrics:
+  statsd: "100.100.123.123:8125" # It's recommended to use IP address instead of domain name
+  # prefix: "sentry.relay" # Adjust this to your needs, default is "sentry.relay"
+  # sample_rate: 1.0 # Adjust this to your needs, default is 1.0
+  # `periodic_secs` is the interval for periodic metrics emitted from Relay.
+  # Setting it to `0` seconds disables the periodic metrics.
+  # periodic_secs: 5
+```
+
+### Symbolicator
+
+You can configure Symbolicator to send metrics to Statsd server by configuring your `symbolicator/config.yml` file:
+
+```yaml
+# The rest of your config.yml file
+metrics:
+  statsd: "100.100.123.123:8125" # It's recommended to use IP address instead of domain name
+  prefix: "sentry.symbolicator" # Adjust this to your needs, default is "symbolicator"
+```
+
+## Sentry dependencies
+
+We don't provide configurations for Sentry's dependencies such as PostgreSQL, Kafka, Redis, Memcached and ClickHouse that are bundled with the Docker Compose file. You will need to provide monitoring configuration for those service yourself adjusted to your needs.