Skip to content

adjust monitoring for container host/server container #3750

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
- Improved documentation about monitoring in Administration
Guide
- Added new backup/restore implementation
- Added new SSL certificate usage using podman secrets
- Added descriptions about various containers used
Expand Down
112 changes: 98 additions & 14 deletions modules/administration/pages/monitoring.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -97,9 +97,9 @@ The Prometheus software is also available for {productname} Proxy and {productna

[IMPORTANT]
====
* To access a shell inside the Server container run [literal]``mgrctl term`` on the container host.
* To access a shell inside the {productname} Server container run [literal]``mgrctl term`` on the container host, or to execute one command run [literal]``mgrctl exec <options> -- <command>``.

* To copy files from inside the container use [literal]``mgrctl cp``.
* To copy files from inside the container to the container host use [literal]``mgrctl cp``.
====

[NOTE]
Expand All @@ -126,20 +126,31 @@ For more information about the monitoring formulas, see xref:specialized-guides:

.Procedure: Manually Installing and Configuring Prometheus
. On the monitoring server, install the [package]``golang-github-prometheus-prometheus`` package:

+

[source,shell]
----
zypper in golang-github-prometheus-prometheus
----

. Enable the Prometheus service:

+

[source,shell]
----
systemctl enable --now prometheus
----

. Check that the Prometheus interface loads correctly.
In your browser, navigate to the URL of the server where Prometheus is installed, on port 9090 (for example, [literal]``http://example.com:9090``).
In your browser, navigate to the URL of the server where Prometheus is installed, on port 9090 (for example, [literal]``http://example.com:9090``).

. Open the configuration file at [path]``/etc/prometheus/prometheus.yml`` and add this configuration information.
Replace `server.url` with your {productname} server URL and adjust `username` and `password` fields to match your {productname} credentials.
Replace `server.url` with your {productname} server URL and adjust `username` and `password` fields to match your {productname} credentials.

+

----
# {productname} self-health metrics
scrape_configs:
Expand Down Expand Up @@ -188,9 +199,14 @@ scrape_configs:
- source_labels: [__meta_uyuni_scheme]
target_label: __scheme__
----

. Save the configuration file.

. Restart the Prometheus service:

+

[source,shell]
----
systemctl restart prometheus
----
Expand Down Expand Up @@ -230,15 +246,23 @@ You can choose which dashboards to provision in the formula configuration page.
.Procedure: Manually Installing Grafana

. Install the [package]``grafana`` package:

+

[source,shell]
----
zypper in grafana
----

. Enable the Grafana service:

+

[source,shell]
----
systemctl enable --now grafana-server
----

. In your browser, navigate to the URL of the server where Grafana is installed, on port 3000 (for example, [literal]``http://example.com:3000``).
. On the login page, enter ``admin`` for username and password.
. Click btn:[Log in].
Expand All @@ -253,7 +277,9 @@ systemctl enable --now grafana-server
. To import a dashboard click the btn:[+] icon in the side menu, and then click btn:[Import].
. For {productname} server overview load the dashboard ID: ``17569``.
. For {productname} clients overview load the dashboard ID: ``17570``.

+

image::monitoring_grafana_example.png[scaledwidth=80%]

[NOTE]
Expand Down Expand Up @@ -454,7 +480,7 @@ Every salt_queue value has a label named `queue` with the queue number as value.

=== Monitoring Managed Systems

Prometheus metrics exporters can be installed and configured on Salt clients using formulas.
Prometheus metrics exporters can be installed and configured on {salt} clients using formulas.
The packages are available from the {productname} client tools channels, and can be enabled and configured directly in the {productname} {webui}.

These exporters can be installed on managed systems:
Expand All @@ -468,7 +494,7 @@ These exporters can be installed on managed systems:

[NOTE]
====
On {sle-micro}, only the Node exporter and the Blackbox exporter are available.
On {sl-micro}, only the Node exporter and the Blackbox exporter are available.
====

When you have the exporters installed and configured, you can start using Prometheus to collect metrics from the monitored systems.
Expand Down Expand Up @@ -579,15 +605,73 @@ This section demonstrates how to generate client/server certificates for Prometh

.Procedure: Creating server/client TLS certificate

. On the {productname} Server, at the command prompt, run following command:
. At the command prompt of the {productname} container host, as root, run the following commands:

+

--

.. To generate certificate files, run the following command.

+

Ensure that the [systemitem]``set-cname`` parameter is the fully qualified domain name (FQDN) of your {salt} client.
You can use the the [systemitem]``set-cname`` parameter multiple times if you require multiple aliases:

+

[source, shell]
----
mgrctl exec -ti -- mgr-ssl-tool --gen-server --dir="/root/ssl-build" --set-country="COUNTRY" \
--set-state="STATE" --set-city="CITY" --set-org="ORGANIZATION" \
--set-org-unit="ORGANIZATION UNIT" --set-email="[email protected]" \
--set-hostname="minion.example.com" --set-cname="minion.example.com" --no-rpm
----

+

Resulting in:

+

----
Generating the web server's SSL private key: /root/ssl-build/minion/server.key
Generating web server's SSL certificate request: /root/ssl-build/minion/server.csr
Generating/signing web server's SSL certificate: server.crt
----

.. Copy [path]``server.crt`` and [path]``server.key`` files from the server container to the host:

+

----
rhn-ssl-tool --gen-server --dir="/root/ssl-build" --set-country="COUNTRY" \
--set-state="STATE" --set-city="CITY" --set-org="ORGANIZATION" \
--set-org-unit="ORGANIZATION UNIT" --set-email="[email protected]" \
--set-hostname="minion.example.com" --set-cname="minion.example.com" --no-rpm
mgrctl cp server:/root/ssl-build/minion/server.key server.key
mgrctl cp server:/root/ssl-build/minion/server.crt server.crt
----
Ensure that the [systemitem]``set-cname`` parameter is the fully qualified domain name (FQDN) of your Salt client.
You can use the the [systemitem]``set-cname`` parameter multiple times if you require multiple aliases.

. Copy ``server.crt`` and ``server.key`` files to the Salt minion and provide read access for ``prometheus`` user.
.. Copy [path]``server.crt`` and [path]``server.key`` files from the host to the monitoring client:

+

----
ssh minion.example.com 'mkdir /etc/ssl/mlm-server-certs'
scp /root/server.* minion.example.com:/etc/ssl/mlm-server-certs
ssh minion.example.com 'chmod go+r /etc/ssl/mlm-server-certs/server.*; ls -la /etc/ssl/mlm-server-certs'
----

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before start with the configuration of formulas, I'm looking into what I wrote in my card, and I see this section:


From the minion:

suma-bv-50-min-sles15sp4:/etc/ssl # mkdir suma-server-certs
suma-bv-50-min-sles15sp4:/etc/ssl # cd suma-server-certs/
suma-bv-50-min-sles15sp4:/etc/ssl/suma-server-certs # cp /root/server.* .
suma-bv-50-min-sles15sp4:/etc/ssl/suma-server-certs # chmod go+r server.*
suma-bv-50-min-sles15sp4:/etc/ssl/suma-server-certs # ls -la
total 16
drwxr-xr-x 2 root root   42 jul  5 11:46 .
drwxr-xr-x 6 root root  183 jul  5 11:45 ..
-rw-r--r-- 1 root root 8335 jul  5 11:46 server.crt
-rw-r--r-- 1 root root 3268 jul  5 11:46 server.key

And we use that path on the minion when we pass the server certificate and key into the Prometheus Formula of the minion.

I would say that's missing right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe there is something missing. I copied the files from inside the server container to the server container host:

mgrctl cp server:/root/ssl-build/minion/server.key server.key
mgrctl cp server:/root/ssl-build/minion/server.crt server.crt

Then with ssh to the monitoring minion (= the client with Prometheur/Grafana)

ssh minion.example.com 'mkdir /etc/ssl/mlm-server-certs'
scp /root/server.* minion.example.com:/etc/ssl/mlm-server-certs
ssh minion.example.com 'chmod go+r /etc/ssl/mlm-server-certs/server.*; ls -la /etc/ssl/mlm-server-certs'

What should come next is unclear to me. I think you'll now use the formula to prepare clients that should be monitored? I'm not sure what your hostnames stand for.

--

. To configure {salt} formulars, enter the directory names specified in the previous steps.

+

--

.. formular server
Server Certificate /etc/ssl/mlm-server-certs/server.crt
Server Key /etc/ssl/mlm-server-certs/server.key


.. formular minion

--
Loading