Prometheus Monitoring

A monitoring solution for NicLabs projects using Prometheus for metric storage, Grafana for visualization, and AlertManager with a Telegram Bot for alerts.

This repository also contains scripts and configuration files for the following exporters:

• cAdvisor
• Blackbox Exporter
• Nginx Exporter
• Node Exporter
• Postgres Exporter

Dependencies

The monitoring server and all the exporters are run using Docker.

Logic

The monitor uses the following logical architecture:

The modules for data storage, visualization and alerting are run as separate containers on the same host machine.

The exporters can be run on however many machines are necessary, they're only required to be accessible from the Prometheus Server container through an HTTP GET request.

Deploy Instructions

Exporters

Each exporter has a directory under the Exporters directory. Each individual directory has a run.sh script to run, start, stop, restart and delete the appropriate exporter's container, as well as it's configuration files when necessary.

You can also control many exporters at once using the run.sh script located directly under the Exporters directory. This script has the following usage syntax:

./run.sh [run | start | stop | restart | delete] [exporter list]

Where exporter list is a comma separated list which can contain cadvisor,blackbox,nginx,node,postgres, or simply all if you want to use all of them.

For example, if you want to run the nginx and Postgres exporters simply run:

./run.sh run nginx,postgres

Server

The server is controlled by the run.sh script from the Server directory.

To deploy the server:

1. Configure the AlertManager:
   1. Configure the telegram bot editing the config.yaml file under prometheus_bot and adding the correct telegram token.
   2. Edit the config.yml file under AlertManager and add the bot's IP address and telegram chatID.
2. Configure the Grafana Server:
   1. Go to the dashboards directory and place a separate json file for every dashboard you wish to automatically import to grafana. The repository contains many example dashboards which use the repository's exporters (based on the dashboards from https://github.com/stefanprodan/dockprom), you should delete the ones you don't want before running the monitor.
3. Configure the Prometheus Server:
   1. Edit the prometheus.yml under the prometheus directory to set which exporters prometheus will scrape for metrics. The repository contains an example configuration you can use to scrape every exporter from the Exporters directory.
   2. Edit the .rules files under the prometheus directory to define under which conditions the AlertManager should be notified. You can use the files included in this repository as an example.
4. Run the run.sh under the Server directory to manage the server's containers.

The run.sh script has the following usage:

./run.sh [run | start | stop | restart | delete]

Use:

./run.sh run

To create and start all the containers. This will create a storage directory which contains the prometheus database.