Enable persistent journaling

ansible/.gitignore

@@ -98,3 +98,5 @@ roles/*
 !roles/eessi/**
 !roles/topology/
 !roles/topology/**
+!roles/journald/
+!roles/journald/**

ansible/bootstrap.yml

@@ -28,6 +28,15 @@
           to update these variable names. ** NB: The actual secrets will not be changed.**
       when: "'secrets_openhpc_' in (hostvars[inventory_hostname] | join)"
 
+- hosts: journald
+  gather_facts: false
+  tags:
+    - logging
+    - journald
+  tasks:
+    - import_role:
+        name: journald
+
 - hosts: resolv_conf
   become: yes
   gather_facts: false

ansible/roles/compute_init/README.md

@@ -40,6 +40,7 @@ it also requires an image build with the role name added to the
 | hooks/pre.yml            | ?                       | None at present                 | n/a                 |
 | validate.yml             | n/a                     | Not relevant during boot        | n/a                 |
 | bootstrap.yml            | (wait for ansible-init) | Not relevant during boot        | n/a                 |
+| bootstrap.yml            | journald                | Fully supported                 | No                  |
 | bootstrap.yml            | resolv_conf             | Fully supported                 | No                  |
 | bootstrap.yml            | etc_hosts               | Fully supported                 | No                  |
 | bootstrap.yml            | chrony                  | Fully supported                 | No                  |

ansible/roles/journald/README.md

@@ -0,0 +1,4 @@
+# journald
+
+This role is used to configue journald. Please see the [role
+defaults](defaults/main.yml) for a full list of configuration options.

ansible/roles/journald/defaults/main.yml

@@ -0,0 +1,4 @@
+---
+# Journald storage. One of: volatile, persistent, auto, or none. Defaults to
+# `persistent`.
+journald_storage: persistent

ansible/roles/journald/tasks/main.yml

@@ -0,0 +1,32 @@
+---
+- name: Create /var/log/journal
+  become: true
+  file:
+    path: /var/log/journal
+    owner: "root"
+    group: "systemd-journal"
+    mode: 02755
+    state: directory
+  notify: Flush journal to disk
+  when: journald_storage == "persistent"
+
+- name: Ensure journald drop in directory exists
+  file:
+    path: "/etc/systemd/journald.conf.d/"
+    owner: "root"
+    group: "root"
+    mode: 0770
+    state: directory
+  become: true
+
+- name: Ensure journald.conf overrides are set
+  copy:
+    content: |
+       [Journal]
+       Storage={{ journald_storage }}
+    dest: /etc/systemd/journald.conf.d/ansible-slurm-appliance.conf
+    owner: root
+    group: root
+    mode: 0660
+  become: true
+  notify: Restart journald

docs/experimental/isolated-clusters.md

@@ -44,6 +44,7 @@ See above for definition of "Default" features. In the "Isolated?" column:
 | gateway               | n/a | n/a - build only | 
 | grafana               | Y | Y | 
 | hpctests              | Y | Y | 
+| journald              | Y | Y |
 | k3s_agent             | - | ? | 
 | k3s_server            | - | ? | 
 | k9s                   | - | ? | 

docs/monitoring-and-logging.md

@@ -301,6 +301,9 @@ slurm-stats is configured `slurm-stats` role in the [slurm_openstack_tools colle
 The `slurm_stats` group controls the placement of the `slurm_stats` service.
 This should be configured to be a group with a single host. That host must be co-located on the same host as the `filebeat` service that scrapes its output.
 
+## Logging configuration
 
+### Journald
 
-
+The [journald](../ansible/roles/journald/README.md) role is used to customise
+journald configuration.

environments/common/inventory/groups

@@ -26,6 +26,9 @@ login
 openhpc
 additional
 
+[journald]
+# Hosts where journald should be configured. See ansible/roles/journald/README.md.
+
 [builder]
 # Do not add hosts here manually - used as part of Packer image build pipeline. See packer/README.md.
 

environments/site/inventory/groups

@@ -81,6 +81,10 @@ cluster
 # Hosts to recompile Slurm for - allows supporting Slurm autodetection method 'nvml'
 cuda
 
+[journald:children]
+# Hosts where journald should be configured. See ansible/roles/journald/README.md.
+cluster
+
 [eessi:children]
 # Hosts on which EESSI stack should be configured
 openhpc