docs: Explain how to work with auto-generated keys; Restructure to clarify how to specify keys for authoritative fields.

docs/src/user-guide/index.md

@@ -14,10 +14,10 @@ A guide to viewing log files in the log viewer.
 :::
 
 :::{grid-item-card}
-:link: format-struct-logs-overview
-Formatting structured logs
+:link: struct-logs/index
+Working with structured logs
 ^^^
-A guide to formatting structured (e.g. JSON) logs as plain text.
+Guides and reference docs for working with structured logs.
 :::
 
 :::{grid-item-card}
@@ -37,12 +37,12 @@ quick-start
 
 :::{toctree}
 :hidden:
-:caption: Formatting structured logs
+:caption: Working with structured logs
 :glob:
 
-format-struct-logs-overview
-format-struct-logs-syntax
-format-struct-logs-formatters
+struct-logs/index
+struct-logs/format/index
+struct-logs/specifying-keys
 :::
 
 :::{toctree}

docs/src/user-guide/format-struct-logs-overview.md → docs/src/user-guide/struct-logs/format/index.md

@@ -1,4 +1,4 @@
-# Overview
+# Formatting as plain text
 
 The log viewer can format structured (e.g. JSON) logs as plain text using a format string. The
 format string allows you to select which fields to include and how they should be formatted. You can
@@ -33,5 +33,12 @@ The formatted log would appear as:
 ```
 
 For reference docs, see:
-* [Format string syntax](format-struct-logs-syntax)
-* [Formatters](format-struct-logs-formatters)
+* [Format string syntax](syntax)
+* [Formatters](formatters)
+
+:::{toctree}
+:hidden:
+
+syntax
+formatters
+:::

docs/src/user-guide/format-struct-logs-syntax.md → docs/src/user-guide/struct-logs/format/syntax.md

@@ -16,22 +16,14 @@ backslash:
 Each field placeholder is enclosed in braces (`{` and `}`) and has the following form, consisting of
 three components:
 ```
-{<field-name>[:<formatter-name>[:<formatter-options>]]}
+{<field-key>[:<formatter-name>[:<formatter-options>]]}
 ```
 
-### field-name (required)
+### field-key (required)
 Defines the key of the field whose value should replace the placeholder.
 
-* Nested fields can be specified using periods (`.`) to denote hierarchy.
-  * E.g., the field `{"a:" {"b": 0}}` may be denoted by `a.b`.
-* Field names can contain any character, except the following characters must be escaped with a
-  backslash:
-  * `.`
-  * `$`
-  * `{`
-  * `}`
-  * `:`
-  * `\`
+:::{include} ../key-syntax.md
+:::
 
 ### formatter-name (optional)
 The name of the formatter to apply to the value before inserting it into the string.
@@ -43,6 +35,8 @@ The name of the formatter to apply to the value before inserting it into the str
   * `:`
   * `\`
 
+For a list of currently supported formatters, see [Formatters](formatters).
+
 ### formatter-options (optional)
 Defines any options for the formatter denoted by `formatter-name`.
 
@@ -63,28 +57,31 @@ Every format string contains an implicit trailing newline so that each formatted
 a newline.
 
 ## Examples
-Consider the following log event:
-```
+
+### Formatting JSON logs events
+
+Consider the following JSON log event:
+```json
 {
-  "@timestamp": 1427153388942,
+  "ts": 1427153388942,
   "level": "INFO",
   "thread": 0,
   "latency": {
     "msecs": 56400,
-    "secs": 56.4,
+    "secs": 56.4
   },
-  "an.odd.key{name}": "org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties"
+  "@an.odd.key{name}": "org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties"
 }
 ```
 
-We can format this using the following YScope format string:
+We can format this using the following format string:
 
 ```
-{@timestamp:timestamp:YYYY-MM-DD HH\:mm\:ss.SSS} {level} \{{thread}\} latency={latency.secs:round} {an\.odd\.key\{name\}}
+{ts:timestamp:YYYY-MM-DD HH\:mm\:ss.SSS} {level} \{{thread}\} latency={latency.secs:round} {\@an\.odd\.key\{name\}}
 ```
 
-* In the first placeholder, we have the field name `@timestamp`, a formatter called `timestamp`, and
-  the formatter's options which are a date format string.
+* In the first placeholder, we have the field key `ts`, a formatter called `timestamp`, and the
+  formatter's options which are a date format string.
 * The second and third placeholders simply stringify the values of the given fields.
 * The fourth placeholder uses the `round` formatter to round a nested field's value; this
   placeholder doesn't specify any formatter options, so the defaults will be used.
@@ -95,4 +92,40 @@ The formatted string will be:
 2015-03-23 19:29:48.942 INFO {0} latency=56 org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
 ```
 
-For a list of currently supported formatters, see [Formatters](format-struct-logs-formatters).
+### Formatting kv-pair IR log events
+
+Consider the following kv-pair IR log event:
+
+:::{note}
+In the example below, for simplicity, we render the log event as JSON with the auto-generated
+kv-pairs under the `auto-generated` key, and the user-generated kv-pairs under the `user-generated`
+key, but these keys don't exist in the log event.
+:::
+
+```json
+{
+  "auto-generated": {
+    "ts": 1741371422000
+  },
+  "user-generated": {
+    "message": "Callback registered to fire in 5 seconds:",
+    "ts": 1741371427000
+  }
+}
+```
+
+We can format this using the following format string:
+
+```
+{@ts:timestamp} {message} {ts:timestamp}
+```
+
+* In the first placeholder, we have the auto-generated field key `@ts`. The `@` prefix specifies
+  that the field is from the auto-generated namespace.
+* The second and third placeholders refer to the `message` and `ts` fields in the user-generated
+  namespace.
+
+The formatted string will be:
+```
+2025-03-07T18:17:02Z Callback registered to fire in 5 seconds: 2025-03-07T18:17:07Z
+```

docs/src/user-guide/struct-logs/index.md

@@ -0,0 +1,21 @@
+# Overview
+
+The following are guides for working with structured logs.
+
+::::{grid} 1 1 2 2
+:gutter: 2
+
+:::{grid-item-card}
+:link: format/index
+Formatting as plain text
+^^^
+A guide to formatting logs as plain text with a format string.
+:::
+
+:::{grid-item-card}
+:link: specifying-keys
+Specifying keys
+^^^
+A guide on how to specify keys in YScope syntax.
+:::
+::::

docs/src/user-guide/struct-logs/key-syntax.md

@@ -0,0 +1,14 @@
+* Nested keys can be specified using periods (`.`) to denote hierarchy.
+  * E.g., the field `{"a:" {"b": 0}}` may be denoted by `a.b`.
+* Auto-generated keys in a [Key-Value Pair IR Stream][kv-pair-ir] can be specified by using `@` as
+  a prefix.
+  * E.g., the auto-generated key `ts` would be specified as `@ts`.
+* Keys can contain any character, except the following characters must be escaped with a backslash:
+  * `.`
+  * `@`
+  * `{`
+  * `}`
+  * `:`
+  * `\`
+
+[kv-pair-ir]: https://docs.yscope.com/clp/main/dev-guide/design-key-value-pair-ir-stream.html

docs/src/user-guide/struct-logs/specifying-keys.md

@@ -0,0 +1,12 @@
+# Specifying keys
+
+Viewing structured logs requires specifying keys for:
+* The format string
+* Authoritative fields such as the log level and timestamp
+
+Both options are found the settings dialog ({fas}`gear`).
+
+## Syntax
+
+:::{include} key-syntax.md
+:::

src/components/modals/SettingsModal/SettingsDialog.tsx

@@ -43,18 +43,19 @@
     {
         helperText: (
             <span>
-                [JSON] Format string for formatting a JSON log event as plain text. See the
+                [Structured] Format string for formatting a structured log event as plain text.
+                Leave blank to display the entire log event. See
                 {" "}
                 <Link
-                    href={"https://docs.yscope.com/yscope-log-viewer/main/user-guide/format-struct-logs-overview.html"}
+                    href={"https://docs.yscope.com/yscope-log-viewer/main/user-guide/struct-logs/format/index.html"}
                     level={"body-sm"}
                     rel={"noopener"}
                     target={"_blank"}
                 >
-                    format string syntax docs
+                    here
                 </Link>
                 {" "}
-                or leave this blank to display the entire log event.
+                for syntax.
             </span>
         ),
         initialValue: getConfig(CONFIG_KEY.DECODER_OPTIONS).formatString,
@@ -63,14 +64,44 @@
         type: "text",
     },
     {
-        helperText: "[JSON] Key to extract the log level from.",
+        helperText: (
+            <span>
+                [Structured] Key that maps to each log event's log level. See
+                {" "}
+                <Link
+                    href={"https://docs.yscope.com/yscope-log-viewer/main/user-guide/struct-logs/specifying-keys.html"}
+                    level={"body-sm"}
+                    rel={"noopener"}
+                    target={"_blank"}
+                >
+                    here
+                </Link>
+                {" "}
+                for syntax.
+            </span>
+        ),
         initialValue: getConfig(CONFIG_KEY.DECODER_OPTIONS).logLevelKey,
         label: "Decoder: Log level key",
         name: LOCAL_STORAGE_KEY.DECODER_OPTIONS_LOG_LEVEL_KEY,
         type: "text",
     },
     {
-        helperText: "[JSON] Key to extract the log timestamp from.",
+        helperText: (
+            <span>
+                [Structured] Key that maps to each log event's timestamp. See
+                {" "}
+                <Link
+                    href={"https://docs.yscope.com/yscope-log-viewer/main/user-guide/struct-logs/specifying-keys.html"}
+                    level={"body-sm"}
+                    rel={"noopener"}
+                    target={"_blank"}
+                >
+                    here
+                </Link>
+                {" "}
+                for syntax.
+            </span>
+        ),
         initialValue: getConfig(CONFIG_KEY.DECODER_OPTIONS).timestampKey,
         label: "Decoder: Timestamp key",
         name: LOCAL_STORAGE_KEY.DECODER_OPTIONS_TIMESTAMP_KEY,