Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS-9171] Add unresolvable behavior template syntax #27299

Open
wants to merge 2 commits into
base: may/2024-q4-obs-pipelines
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 16 additions & 12 deletions content/en/observability_pipelines/destinations/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,18 +39,22 @@

## Template syntax

Logs are often stored in separate indexes based on log data, such as the service or environment the logs are coming from or another log attribute. In Observability Pipelines, you can use template syntax to route your logs to different indexes based on specific log fields. The following destinations and fields support template syntax:

| Destination | Fields that support template syntax |
| ----------------- | -------------------------------------|
| Amazon Opensearch | Index |
| Amazon S3 | Prefix |
| Azure Blob | Prefix |
| Elasticsearch | Source type |
| Google Chronicle | Log type |
| Google Cloud | Prefix |
| Opensearch | Index |
| Splunk HEC | Index<br>Source type |
Logs are often stored in separate indexes based on log data, such as the service or environment the logs are coming from or another log attribute. In Observability Pipelines, you can use template syntax to route your logs to different indexes based on specific log fields.

Check notice on line 42 in content/en/observability_pipelines/destinations/_index.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.sentencelength

Suggestion: Try to keep your sentence length to 25 words or fewer.

When the Observability Pipelines Worker cannot resolve the field with the template syntax, the Worker defaults to a specified behavior for that destination. For example, if you are using the template `{{application_id}}` for the Amazon S3 destination's **Prefix** field, but there isn't an `application_id` field in the log, the Worker creates a folder called `OP_UNRESOLVED_TEMPLATE_LOGS/` and publishes the logs there.

Check notice on line 44 in content/en/observability_pipelines/destinations/_index.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.sentencelength

Suggestion: Try to keep your sentence length to 25 words or fewer.

The following table lists the destinations and fields that support template syntax, and what happens when the Worker cannot resolve the field:

| Destination | Fields that support template syntax | Behavior when the field cannot be resolved |
| ----------------- | -------------------------------------| -----------------------------------------------------------------------------------------------|
| Amazon Opensearch | Index | The Worker creates an index named `datadog-op` and sends the logs there. |
| Amazon S3 | Prefix | The Worker creates a folder named `OP_UNRESOLVED_TEMPLATE_LOGS/` and publishes the logs there. |
| Azure Blob | Prefix | The Worker creates a folder named `OP_UNRESOLVED_TEMPLATE_LOGS/` and publishes the logs there. |
| Elasticsearch | Source type | The Worker creates an index named `datadog-op` and sends the logs there. |
| Google Chronicle | Log type | Defaults to `vector_dev` log type. |
| Google Cloud | Prefix | The Worker creates a folder named `OP_UNRESOLVED_TEMPLATE_LOGS/` and publishes the logs there. |
| Opensearch | Index | The Worker creates an index named `datadog-op` and sends the logs there. |
| Splunk HEC | Index<br>Source type | The Worker publishes the logs to the default index configured in Splunk. |
Comment on lines +49 to +57
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do "send" and "publish" have two different meanings in this case?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yeah, meant to ask eng about that. Asking now.


#### Example

Expand Down
Loading