|
| 1 | +# Setup Monitor System |
| 2 | + |
| 3 | +This article will guide you through the process of setting up the monitor system for ***foyer***. |
| 4 | + |
| 5 | +***foyer*** provides observability for monitoring in 3 aspects: |
| 6 | +- **logging**: Provides logging in trace/debug/info/warn/error levels with [tracing](https://crates.io/crates/tracing) ecosystem. |
| 7 | +- **metrics**: Provides operation counters, operation duration histograms, information gauges with [metrics](https://crates.io/crates/metrics) ecosystem. |
| 8 | +- **tracing**: Provide tail-based tracing for slow operation diagnosis with [fastrace](https://crates.io/crates/fastrace). |
| 9 | + |
| 10 | +For each ecosystem, there are rich surrounding libraries available. The article will only introduce the example of the most basic configuration. For more details, please refer to the document of each ecosystem. |
| 11 | + |
| 12 | +## 1. Setup logging monitoring |
| 13 | + |
| 14 | +***foyer*** uses [tracing](https://crates.io/crates/tracing) ecosystem for logging monitoring. You can configure the filters and subscribers to control the format, content, and the target for logging. |
| 15 | + |
| 16 | +Here is an example to setup logging monitoring to the console, decorate each log entry with line number, and filter the logs with `RUST_LOG` syntax. |
| 17 | + |
| 18 | +Add the tracing-subscriber dependencies. |
| 19 | + |
| 20 | +```toml |
| 21 | +tracing-subscriber = { version = "0.3", features = ["env-filter"] } |
| 22 | +``` |
| 23 | + |
| 24 | +Add the following lines at the start of your project. |
| 25 | + |
| 26 | +```rust |
| 27 | +tracing_subscriber::registry() |
| 28 | + .with(tracing_subscriber::fmt::layer().with_line_number(true)) |
| 29 | + .with(EnvFilter::from_default_env()) |
| 30 | + .init(); |
| 31 | +``` |
| 32 | + |
| 33 | +Then everything is set now! Now, you can run your project directly, or with a customized `RUST_LOG` arguments: |
| 34 | + |
| 35 | +```bash |
| 36 | +RUST_LOG=foyer_storage::large::generic=trace,info ./your-project-with-foyer |
| 37 | +``` |
| 38 | + |
| 39 | +Here is a sample of the outputs: |
| 40 | + |
| 41 | +```plain |
| 42 | +2024-10-27T08:50:08.631742Z INFO foyer_bench: 414: [foyer bench]: jemalloc is enabled. |
| 43 | +2024-10-27T08:50:08.632902Z WARN foyer_storage::store: 482: [store builder]: Setting up small object disk cache options, but only large object disk cache is enabled. |
| 44 | +2024-10-27T08:50:08.639524Z INFO foyer_storage::large::recover: 162: Recovers 0 regions with data, 1600 clean regions, 0 total entries with max sequence as 0, initial reclaim permits is 0. |
| 45 | +2024-10-27T08:50:08.659042Z TRACE foyer_storage::large::generic: 344: EntryAddress { |
| 46 | + region: 1, |
| 47 | + offset: 8192, |
| 48 | + len: 1420, |
| 49 | + sequence: 19, |
| 50 | +} |
| 51 | +2024-10-27T08:50:08.660765Z TRACE foyer_storage::large::generic: 344: EntryAddress { |
| 52 | + region: 0, |
| 53 | + offset: 28672, |
| 54 | + len: 638, |
| 55 | + sequence: 41, |
| 56 | +} |
| 57 | +``` |
| 58 | + |
| 59 | +For more details, please refer to the [document](https://docs.rs/tracing/0.1.40/tracing/) of `tracing`. |
| 60 | + |
| 61 | +## 2. Setup metrics monitoring |
| 62 | + |
| 63 | +***foyer*** uses [metrics](https://crates.io/crates/metrics) ecosystem for metrics monitoring. [metrics](https://crates.io/crates/metrics) works like an adapter, ***foyer*** uses `metrics` to define what metrics to collect, and you can use the libraries from the metrics ecosystem to define how and where to collect them. (e.g. Prometheus, StatsD, NewRelic, etc.) Please refer to the [README](https://github.com/metrics-rs/metrics) and [document](https://github.com/metrics-rs/metrics) |
| 64 | + |
| 65 | +This section will take integrating with *Prometheus* as an example. |
| 66 | + |
| 67 | +To integrate *Prometheus*，the trivial method is using the `metrics-exporter-prometheus` offered by `metrics`. |
| 68 | + |
| 69 | +First, add the dependency to your project: |
| 70 | + |
| 71 | +```toml |
| 72 | +metrics-exporter-prometheus = "0.15" |
| 73 | +``` |
| 74 | + |
| 75 | +Then setup the prometheus exporter. |
| 76 | + |
| 77 | +```rust |
| 78 | +let addr: SocketAddr = "0.0.0.0:19970".parse().unwrap(); |
| 79 | +PrometheusBuilder::new() |
| 80 | + .with_http_listener(addr) |
| 81 | + .set_buckets(&[0.000_001, 0.000_01, 0.000_1, 0.001, 0.01, 0.1, 1.0]) |
| 82 | + .unwrap() |
| 83 | + .install() |
| 84 | + .unwrap(); |
| 85 | +``` |
| 86 | + |
| 87 | +However, some project may have already using lib [`prometheus`](https://crates.io/crates/prometheus) to export data. In this case, it is recommended to use [`metrics-prometheus`](https://crates.io/crates/metrics-prometheus) as an adapter between lib `metrics` and `prometheus`. |
| 88 | + |
| 89 | +First, add `metrics-prometheus` as an dependency to your project: |
| 90 | + |
| 91 | +```toml |
| 92 | +metrics-prometheus = "0.7" |
| 93 | +``` |
| 94 | + |
| 95 | +Then set the registry your project is using with `prometheus` as the target of `metrics` that ***foyer*** uses with `metrics-prometheus`. |
| 96 | + |
| 97 | +```rust |
| 98 | +// For example, `GLOBAL_METRICS_REGISTRY` is the registry your project is using with lib `prometheus`. |
| 99 | + |
| 100 | +metrics_prometheus::Recorder::builder() |
| 101 | + .with_registry(GLOBAL_METRICS_REGISTRY.clone()) |
| 102 | + .build_and_install(); |
| 103 | +``` |
| 104 | + |
| 105 | +Then export the registry like your project does before. |
| 106 | + |
| 107 | +After collecting data with *Prometheus*, you can setup *Grafana* to display the metrics. |
| 108 | + |
| 109 | +Here is an example of some exported metrics. |
| 110 | + |
| 111 | +<div style="text-align: center;"> |
| 112 | + |
| 113 | +  |
| 114 | + |
| 115 | +</div> |
| 116 | + |
| 117 | +## 3. Setup tracing monitoring |
| 118 | + |
| 119 | +***TBC ... ...*** |
0 commit comments