OBSDOCS-3446: Document Splunk sourceType configuration

modules/default-splunk-metadata-key-values.adoc

@@ -58,7 +58,7 @@ a|Audit
 |`_json` or `generic_single_line`
 |`_json` or `generic_single_line`
 |`_json` or `generic_single_line`
-|Determined automatically based on the type of the final event payload.
+|Default is `_json`. If `payloadKey` is set without `sourceType`, the value is `_json` or `generic_single_line` depending on the final event payload structure. You can configure a custom value (static or templated) when `payloadKey` is defined.
 
 
 |`host`

modules/logging-forward-splunk.adoc

@@ -46,6 +46,7 @@ spec:
             key: hecToken
         index: '{.log_type||"undefined"}'
         source: '{.log_source||"undefined"}'
+        sourceType: 'log4j'
         indexedFields: ['.log_type', '.log_source']
         payloadKey: '.kubernetes'
         tuning:
@@ -64,7 +65,13 @@ spec:
 `key`:: Specify the name of the secret that has your HEC token.
 `url`:: Specify the URL, including port, of your Splunk HEC.
 `index`:: Specify the name of the index to send events to. If you do not specify an index, the Splunk server uses its default index. This is an optional field.
-`source`:: Specify the source of events to send to this sink. You can configure dynamic per-event values. This field is optional. If you do not specify a value, the `log_type` and `log_source` values determine the value of the field. For example, see link:https://docs.redhat.com/en/documentation/red_hat_openshift_logging/6.3/html-single/configuring_logging/index#default-splunk-metadata-key-values_configuring-log-forwarding[Default Splunk metadata key values]. 
+`source`:: Specify the source of events to send to this sink. You can configure dynamic per-event values. This field is optional. If you do not specify a value, the `log_type` and `log_source` values determine the value of the field. For example, see link:https://docs.redhat.com/en/documentation/red_hat_openshift_logging/6.3/html-single/configuring_logging/index#default-splunk-metadata-key-values_configuring-log-forwarding[Default Splunk metadata key values].
+`sourceType`:: Specify a Splunk source type to define how incoming data should be parsed and interpreted. You can use Splunk's pretrained source types (such as `log4j`, `apache:access`, or `linux:syslog`) or custom source types configured in your Splunk instance. This field is optional and can only be set when `payloadKey` is also defined. You can configure static values or dynamic per-event values by using template syntax. If you do not specify a value, the source type is `_json`. If you set `payloadKey` without `sourceType`, the source type is either `_json` or `generic_single_line`, depending on the structure of the final event payload.
++
+[IMPORTANT]
+====
+You are responsible for ensuring that the source type matches the log content. The log collector does not validate or interpret the source type value.
+====
 `indexedFields`:: Specify the fields to add to the Splunk index. This field is optional. The index stores the values directly alongside the raw event data, allowing for faster search performance on those fields.
 However, `indexed_fields` fields increase storage use. Use them only for high-value fields that give significant search benefits, for example, large datasets with frequent queries on specific fields.
 You can use complex and nested fields as indexed fields. These are automatically transformed to meet Splunk's requirements.

modules/logging-splunk-sourcetype-pod-labels.adoc

@@ -0,0 +1,105 @@
+// Module included in the following assemblies:
+//
+// * configuring/configuring-log-forwarding.adoc
+
+:_mod-docs-content-type: PROCEDURE
+[id="logging-splunk-sourcetype-pod-labels_{context}"]
+= Configuring Splunk source types with pod labels
+
+[role="_abstract"]
+You can configure the Splunk `sourceType` field dynamically using pod labels. This allows different applications to specify their own source types for better log parsing in Splunk.
+
+.Prerequisites
+* You have installed the {clo}.
+* You have created a secret with your Splunk HEC token.
+* You have labeled your application pods with the source type information.
+
+.Procedure
+
+. Label your application pod with the required Splunk source type:
++
+[source,yaml]
+----
+apiVersion: v1
+kind: Pod
+metadata:
+  name: myapp
+  labels:
+    app: myapp
+    splunk/sourcetype: log4j
+spec:
+  containers:
+    - name: myapp
+      image: myapp:latest
+      ports:
+        - containerPort: 8080
+----
++
+The `splunk/sourcetype` label contains the Splunk source type value.
++
+[NOTE]
+====
+Kubernetes pod label values cannot contain colon characters (`:`). If you need to use a source type with colons (such as `apache:access` or `my:custom:sourcetype`), you must set the `sourceType` field directly in the `ClusterLogForwarder` custom resource with a static value instead of using a pod label.
+====
+
+. Create or edit the `ClusterLogForwarder` custom resource to use the pod label for the source type:
++
+[source,yaml]
+----
+apiVersion: observability.openshift.io/v1
+kind: ClusterLogForwarder
+metadata:
+  name: instance
+  namespace: openshift-logging
+spec:
+  serviceAccount:
+    name: logging-admin
+  outputs:
+    - name: splunk-receiver
+      type: splunk
+      splunk:
+        url: 'https://your.splunk.hec.url:8088'
+        authentication:
+          token:
+            secretName: splunk-secret
+            key: hecToken
+        payloadKey: .message
+        sourceType: '{.kubernetes.labels."splunk/sourcetype"||"generic_single_line"}'
+  pipelines:
+    - name: application-logs
+      inputRefs:
+        - application
+      outputRefs:
+        - splunk-receiver
+----
++
+The `payloadKey` field is required when using `sourceType`. The template syntax in the `sourceType` field reads the `splunk/sourcetype` label from the pod. If the label is not present, the source type defaults to `generic_single_line`.
+
+.Verification
+
+. Check that the `ClusterLogForwarder` custom resource is ready:
++
+[source,terminal]
+----
+$ oc get clusterlogforwarder instance -n openshift-logging -o yaml
+----
++
+The output should show `Ready: True` in the status conditions.
+
+. In your Splunk instance, verify that logs are arriving with the correct source type:
+.. Navigate to *Search and Reporting*.
+.. Run a search query to check the source type:
++
+.Example output
+[source,terminal]
+----
+index=* sourcetype=log4j
+----
++
+Verify that logs from your labeled pods appear in the search results.
+
+[role="_additional-resources"]
+.Additional resources
+
+* link:https://docs.splunk.com/Documentation/Splunk/latest/Data/Listofpretrainedsourcetypes[List of pretrained source types in Splunk documentation]
+* link:https://docs.splunk.com/Documentation/Splunk/latest/Data/Whysourcetypesmatter[Why source types matter in Splunk documentation]