-
Notifications
You must be signed in to change notification settings - Fork 710
cloud: TiDB Cloud Essential and Premium support Prometheus and Grafana integration #22603
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
huoyao1125
wants to merge
35
commits into
pingcap:release-8.5
Choose a base branch
from
huoyao1125:patch-38
base: release-8.5
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+155
−1
Open
Changes from all commits
Commits
Show all changes
35 commits
Select commit
Hold shift + click to select a range
5eb1a46
Create prometheus-grafana-integration-premium
huoyao1125 a03071b
Update prometheus-grafana-integration-premium
huoyao1125 7bcfba2
Apply suggestions from code review
hfxsd 6e051f7
Update TOC-tidb-cloud-premium.md
hfxsd cc5b372
Update TOC-tidb-cloud-essential.md
hfxsd 5f274ba
Merge branch 'release-8.5' into pr/22603
hfxsd a10df44
add .md
hfxsd d145803
Update prometheus-grafana-integration-premium.md
huoyao1125 e343f10
Create prometheus-grafana-integration-essential
huoyao1125 7e7179b
Update prometheus-grafana-integration-premium.md
huoyao1125 1a484e2
Rename prometheus-grafana-integration-essential to prometheus-grafana…
handlerww 5fd148b
Update TOC-tidb-cloud-essential.md
hfxsd ecd7182
Merge branch 'release-8.5' into pr/22603
hfxsd b595974
remove the redundant doc
hfxsd 5655ee1
Update prometheus-grafana-integration.md
hfxsd 60ed052
Revert "Update prometheus-grafana-integration.md"
hfxsd 82f5340
Update TOC-tidb-cloud-essential.md
hfxsd 78f6f9d
Update prometheus-grafana-integration.md
hfxsd 279c213
Update tidb-cloud/prometheus-grafana-integration.md
hfxsd b61da3e
Update prometheus-grafana-integration.md
hfxsd 537cd28
Update TiDB Cloud Prometheus/Grafana docs
hfxsd f2e4f4a
Update prometheus-grafana-integration.md
hfxsd 1d2bc94
Merge branch 'release-8.5' into pr/22603
hfxsd 62cc435
Update prometheus-grafana-integration.md
hfxsd cb23c1d
Update prometheus-grafana-integration.md
hfxsd d955b4c
Update prometheus-grafana-integration.md
huoyao1125 99adc86
Update prometheus-grafana-integration.md
hfxsd c05ad30
Apply suggestions from code review
hfxsd 392a6a6
Update features.md
hfxsd 16c6c67
Merge branch 'patch-38' of https://github.com/huoyao1125/docs into pr…
hfxsd 58dc82b
Update prometheus-grafana-integration.md
hfxsd 1c7ea47
Update prometheus-grafana-integration.md
hfxsd adcbf88
Update prometheus-grafana-integration.md
hfxsd 206fb83
Update tidb-cloud/prometheus-grafana-integration.md
hfxsd 1968d70
Update tidb-cloud/prometheus-grafana-integration.md
qiancai File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,152 @@ | ||
| --- | ||
| title: Integrate TiDB Cloud with Prometheus and Grafana (Preview) | ||
| summary: Learn how to monitor your TiDB Cloud instances with the Prometheus and Grafana integration. | ||
| --- | ||
|
|
||
| # Integrate TiDB Cloud with Prometheus and Grafana (Preview) | ||
|
|
||
| TiDB Cloud provides a [Prometheus](https://prometheus.io/) API endpoint. If you have a Prometheus service, you can monitor key metrics of TiDB Cloud from the endpoint easily. | ||
|
|
||
| This document describes how to configure your Prometheus service to read key metrics from the <CustomContent plan="essential">{{{ .essential }}}</CustomContent><CustomContent plan="premium">{{{ .premium }}}</CustomContent> endpoint and how to view the metrics using [Grafana](https://grafana.com/). | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - To integrate TiDB Cloud with Prometheus, you must have a self-hosted or managed Prometheus service. | ||
|
|
||
| - To set up third-party metrics integration for TiDB Cloud, you must have the `Organization Owner` or `Instance Manager` access in TiDB Cloud. To view the integration page, you need at least the `Project Viewer` or `Instance Viewer` role to access the target <CustomContent plan="essential">{{{ .essential }}} cluster</CustomContent><CustomContent plan="premium">{{{ .premium }}} instance</CustomContent> under your Organization in TiDB Cloud. | ||
|
|
||
| ## Limitation | ||
|
|
||
| - Prometheus and Grafana integrations are not available for [TiDB Cloud Starter](/tidb-cloud/select-cluster-tier.md#starter) clusters. | ||
| - Prometheus and Grafana integrations are not available when the cluster status is **CREATING**, **RESTORING**, **PAUSED**, or **RESUMING**. | ||
|
|
||
| ## Steps | ||
|
|
||
| ### Step 1. Get a `scrape_config` file for Prometheus | ||
|
|
||
| Before configuring your Prometheus service to read metrics of TiDB Cloud, you need to generate a `scrape_config` YAML file in TiDB Cloud first. The `scrape_config` file contains a unique bearer token that allows the Prometheus service to monitor your target <CustomContent plan="essential">cluster</CustomContent><CustomContent plan="premium">instance</CustomContent>. | ||
|
|
||
| <CustomContent plan="essential"> | ||
|
|
||
| 1. In the [TiDB Cloud console](https://tidbcloud.com/), navigate to the [**Clusters**](https://tidbcloud.com/console/clusters) page, and then click the name of your target {{{ .essential }}} cluster to go to its overview page. | ||
| 2. In the left navigation pane, click **Integrations** > **Integration to Prometheus(Preview)**. | ||
| 3. Click **Add File** to generate and show the `scrape_config` file for the current {{{ .essential }}} cluster. | ||
| 4. Make a copy of the `scrape_config` file content for later use. | ||
|
|
||
| </CustomContent> | ||
|
|
||
| <CustomContent plan="premium"> | ||
|
|
||
| 1. In the [TiDB Cloud console](https://tidbcloud.com/), navigate to the [**My TiDB**](https://tidbcloud.com/tidbs) page, and then click the name of your target {{{ .premium }}} instance to go to its overview page. | ||
| 2. In the left navigation pane, click **Settings** > **Integrations** > **Integration to Prometheus(Preview)**. | ||
| 3. Click **Add File** to generate and show the `scrape_config` file for the current {{{ .premium }}} instance. | ||
| 4. Make a copy of the `scrape_config` file content for later use. | ||
|
|
||
| </CustomContent> | ||
|
|
||
| > **Note:** | ||
| > | ||
| > - For security reasons, TiDB Cloud only shows a newly generated `scrape_config` file once. Ensure that you copy the content before closing the file window. | ||
| > - If you forget, delete the `scrape_config` file in TiDB Cloud and generate a new one. To delete a `scrape_config` file, select the file, click **...**, and then click **Delete**. | ||
|
|
||
| ### Step 2. Integrate with Prometheus | ||
|
|
||
| 1. In the monitoring directory specified by your Prometheus service, locate the Prometheus configuration file. | ||
|
|
||
| For example, `/etc/prometheus/prometheus.yml`. | ||
|
|
||
| 2. In the Prometheus configuration file, locate the `scrape_configs` section, and then copy the `scrape_config` file content obtained from TiDB Cloud to the section. | ||
|
|
||
| 3. In your Prometheus service, check **Status** > **Targets** to verify that the new `scrape_config` file has been read. If not, you might need to restart the Prometheus service. | ||
|
|
||
| ### Step 3. Use Grafana GUI dashboards to visualize the metrics | ||
|
|
||
| After your Prometheus service reads metrics from TiDB Cloud, you can use Grafana GUI dashboards to visualize the metrics as follows: | ||
|
|
||
| 1. Download the Grafana dashboard JSON file for <CustomContent plan="essential">{{{ .essential }}}</CustomContent><CustomContent plan="premium">{{{ .premium }}}</CustomContent> from the following link: | ||
|
|
||
| <CustomContent plan="essential"> | ||
|
|
||
| <https://github.com/pingcap/docs/blob/master/tidb-cloud/monitor-prometheus-and-grafana-integration-tidb-cloud-dynamic-tracker-essential.json> | ||
|
|
||
| </CustomContent> | ||
| <CustomContent plan="premium"> | ||
|
|
||
| <https://github.com/pingcap/docs/blob/master/tidb-cloud/monitor-prometheus-and-grafana-integration-tidb-cloud-dynamic-tracker-premium.json> | ||
|
|
||
| </CustomContent> | ||
|
|
||
| 2. [Import this JSON to your own Grafana GUI](https://grafana.com/docs/grafana/v8.5/dashboards/export-import/#import-dashboard) to visualize the metrics. | ||
|
|
||
| > **Note:** | ||
| > | ||
| > If you are already using Prometheus and Grafana to monitor <CustomContent plan="essential">clusters</CustomContent><CustomContent plan="premium">instances</CustomContent> and want to incorporate the newly available metrics, it is recommended that you create a new dashboard instead of directly updating the JSON of the existing one. | ||
|
|
||
| 3. (Optional) Customize the dashboard as needed by adding or removing panels, changing data sources, and modifying display options. | ||
|
|
||
| For more information about how to use Grafana, see [Grafana documentation](https://grafana.com/docs/grafana/latest/getting-started/getting-started-prometheus/). | ||
|
|
||
| ## Best practice for rotating `scrape_config` | ||
|
|
||
| To improve data security, periodically rotate `scrape_config` file bearer tokens. | ||
|
|
||
| 1. Follow [Step 1](#step-1-get-a-scrape_config-file-for-prometheus) to create a new `scrape_config` file for Prometheus. | ||
| 2. Add the content of the new file to your Prometheus configuration file. | ||
| 3. Once you confirm that your Prometheus service can read from TiDB Cloud, remove the content of the old `scrape_config` file from your Prometheus configuration file. | ||
| 4. On the **Integrations** page of your <CustomContent plan="essential">cluster</CustomContent><CustomContent plan="premium">instance</CustomContent>, delete the corresponding old `scrape_config` file to block anyone else from using it to read from the TiDB Cloud Prometheus endpoint. | ||
|
|
||
| ## Metrics available to Prometheus | ||
|
|
||
| Prometheus tracks the following metric data for your <CustomContent plan="essential">cluster</CustomContent><CustomContent plan="premium">instance</CustomContent>. | ||
|
|
||
| <CustomContent plan="essential"> | ||
|
|
||
| > **Note:** | ||
| > | ||
| > {{{ .essential }}} does not support TiCDC components, so the `tidbcloud_changefeed_*` metrics are currently not available. | ||
|
|
||
| | Metric name | Metric type | Labels | Description | | ||
| |:--- |:--- |:--- |:--- | | ||
| | `tidbcloud_db_total_connection` | gauge | `instance_id: <instance id>`<br/>`instance_name: <instance name>` | The number of current connections in your TiDB server | | ||
| | `tidbcloud_db_active_connections` | gauge | `instance_id: <instance id>`<br/>`instance_name: <instance name>` | The number of active connections | | ||
| | `tidbcloud_db_disconnections` | gauge | `result: Error\|...`<br/>`instance_id: <instance id>`<br/>`instance_name: <instance name>` | The number of clients disconnected by connection result | | ||
| | `tidbcloud_db_database_time` | gauge | `sql_type: Select\|Insert\|...`<br/>`instance_id: <instance id>`<br/>`instance_name: <instance name>` | A time model statistic that represents the sum of all processes' CPU consumption plus the sum of non-idle wait time | | ||
| | `tidbcloud_db_query_per_second` | gauge | `type: Select\|Insert\|...`<br/>`instance_id: <instance id>`<br/>`instance_name: <instance name>` | The number of SQL statements executed per second, counted according to statement types | | ||
| | `tidbcloud_db_failed_queries` | gauge | `type: planner:xxx\|executor:2345\|...`<br/>`instance_id: <instance id>`<br/>`instance_name: <instance name>` | The statistics of error types (for example, syntax errors, primary key conflicts) occurred when executing SQL statements per second | | ||
| | `tidbcloud_db_command_per_second` | gauge | `type: Query\|Ping\|...`<br/>`instance_id: <instance id>`<br/>`instance_name: <instance name>` | The number of commands processed by TiDB per second | | ||
| | `tidbcloud_db_queries_using_plan_cache_ops` | gauge | `instance_id: <instance id>`<br/>`instance_name: <instance name>` | The statistics of queries hitting the Execution Plan Cache per second | | ||
| | `tidbcloud_db_average_query_duration` | gauge | `sql_type: Select\|Insert\|...`<br/>`instance_id: <instance id>`<br/>`instance_name: <instance name>` | The duration between the time a network request is sent to TiDB and returned to the client | | ||
| | `tidbcloud_db_transaction_per_second` | gauge | `type: Commit\|Rollback\|...`<br/>`txn_mode: optimistic\|pessimistic`<br/>`instance_id: <instance id>`<br/>`instance_name: <instance name>` | The number of transactions executed per second | | ||
| | `tidbcloud_db_row_storage_used_bytes` | gauge | `instance_id: <instance id>`<br/>`instance_name: <instance name>` | The row-based storage size of the cluster in bytes | | ||
| | `tidbcloud_db_columnar_storage_used_bytes` | gauge | `instance_id: <instance id>`<br/>`instance_name: <instance name>` | The columnar storage size of the cluster in bytes. Returns 0 if TiFlash is not enabled. | | ||
| | `tidbcloud_resource_manager_resource_request_unit_total` | gauge | `instance_id: <instance id>`<br/>`instance_name: <instance name>` | The total Request Units (RU) consumed.| | ||
|
|
||
| </CustomContent> | ||
|
|
||
| <CustomContent plan="premium"> | ||
|
|
||
hfxsd marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| | Metric name | Metric type | Labels | Description | | ||
| |:--- |:--- |:--- |:--- | | ||
| | `tidbcloud_db_total_connection` | gauge | `instance_id: <instance id>`<br/>`instance_name: <instance name>` | The number of current connections in your TiDB server | | ||
| | `tidbcloud_db_active_connections` | gauge | `instance_id: <instance id>`<br/>`instance_name: <instance name>` | The number of active connections | | ||
| | `tidbcloud_db_disconnections` | gauge | `result: Error\|...`<br/>`instance_id: <instance id>`<br/>`instance_name: <instance name>` | The number of clients disconnected by connection result | | ||
| | `tidbcloud_db_database_time` | gauge | `sql_type: Select\|Insert\|...`<br/>`instance_id: <instance id>`<br/>`instance_name: <instance name>` | A time model statistic that represents the sum of all processes' CPU consumption plus the sum of non-idle wait time | | ||
| | `tidbcloud_db_query_per_second` | gauge | `type: Select\|Insert\|...`<br/>`instance_id: <instance id>`<br/>`instance_name: <instance name>` | The number of SQL statements executed per second, counted according to statement types | | ||
| | `tidbcloud_db_failed_queries` | gauge | `type: planner:xxx\|executor:2345\|...`<br/>`instance_id: <instance id>`<br/>`instance_name: <instance name>` | The statistics of error types (for example, syntax errors, primary key conflicts) occurred when executing SQL statements per second | | ||
| | `tidbcloud_db_command_per_second` | gauge | `type: Query\|Ping\|...`<br/>`instance_id: <instance id>`<br/>`instance_name: <instance name>` | The number of commands processed by TiDB per second | | ||
| | `tidbcloud_db_queries_using_plan_cache_ops` | gauge | `instance_id: <instance id>`<br/>`instance_name: <instance name>` | The statistics of queries hitting the Execution Plan Cache per second | | ||
| | `tidbcloud_db_average_query_duration` | gauge | `sql_type: Select\|Insert\|...`<br/>`instance_id: <instance id>`<br/>`instance_name: <instance name>` | The duration between the time a network request is sent to TiDB and returned to the client | | ||
| | `tidbcloud_db_transaction_per_second` | gauge | `type: Commit\|Rollback\|...`<br/>`txn_mode: optimistic\|pessimistic`<br/>`instance_id: <instance id>`<br/>`instance_name: <instance name>` | The number of transactions executed per second | | ||
| | `tidbcloud_db_row_storage_used_bytes` | gauge | `instance_id: <instance id>`<br/>`instance_name: <instance name>` | The row-based storage size of the cluster in bytes | | ||
| | `tidbcloud_db_columnar_storage_used_bytes` | gauge | `instance_id: <instance id>`<br/>`instance_name: <instance name>` | The columnar storage size of the cluster in bytes. | | ||
| | `tidbcloud_resource_manager_resource_request_unit_total` | gauge | `instance_id: <instance id>`<br/>`instance_name: <instance name>` | The total Request Units (RU) consumed. | | ||
| | `tidbcloud_changefeed_latency` | gauge | `changefeed: <changefeed-id>`<br/>`instance_id: <instance id>`<br/>`instance_name: <instance name>` | The data replication latency between the upstream and the downstream of a changefeed | | ||
| | `tidbcloud_changefeed_status` | gauge | `changefeed: <changefeed-id>`<br/>`instance_id: <instance id>`<br/>`instance_name: <instance name>` | Changefeed status:<br/>`-1`: Unknown<br/>`0`: Normal<br/>`1`: Warning<br/>`2`: Failed<br/>`3`: Stopped<br/>`4`: Finished<br/>`6`: Warning<br/>`7`: Other | | ||
|
|
||
| </CustomContent> | ||
|
|
||
| ## FAQ | ||
|
|
||
| - Why does the same metric have different values on Grafana and the TiDB Cloud console at the same time? | ||
|
|
||
| Grafana and TiDB Cloud use different aggregation calculation logic, so the displayed aggregated values might differ. You can adjust the `mini step` configuration in Grafana to get more fine-grained metric values. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.