Skip to content

[DOCS-10730] Add Obs Pipelines 2.7 components #30454

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 13 additions & 8 deletions config/_default/menus/main.en.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5456,46 +5456,51 @@ menu:
url: /observability_pipelines/destinations/google_cloud_storage/
parent: observability_pipelines_destinations
weight: 1109
- name: HTTP Client
url: observability_pipelines/destinations/http_client/
parent: observability_pipelines_destinations
identifier: observability_pipelines_http_client
weight: 1110
- name: Microsoft Sentinel
identifier: observability_pipelines_microsoft_sentinel
url: /observability_pipelines/destinations/microsoft_sentinel/
parent: observability_pipelines_destinations
weight: 1110
weight: 1111
- name: New Relic
identifier: observability_pipelines_new_relic
url: /observability_pipelines/destinations/new_relic/
parent: observability_pipelines_destinations
weight: 1111
weight: 1112
- name: OpenSearch
url: observability_pipelines/destinations/opensearch
parent: observability_pipelines_destinations
identifier: observability_pipelines_opensearch
weight: 1112
weight: 1113
- name: SentinelOne
url: observability_pipelines/destinations/sentinelone
parent: observability_pipelines_destinations
identifier: observability_pipelines_sentinelone
weight: 1113
weight: 1114
- name: Socket
url: observability_pipelines/destinations/socket
parent: observability_pipelines_destinations
identifier: observability_pipelines_socket
weight: 1114
weight: 1115
- name: Splunk HEC
url: observability_pipelines/destinations/splunk_hec
parent: observability_pipelines_destinations
identifier: observability_pipelines_splunk_hec
weight: 1115
weight: 1116
- name: Sumo Logic Hosted Collector
url: observability_pipelines/destinations/sumo_logic_hosted_collector
parent: observability_pipelines_destinations
identifier: observability_pipelines_sumo_logic_hosted_collector
weight: 1116
weight: 1117
- name: Syslog
url: observability_pipelines/destinations/syslog
parent: observability_pipelines_destinations
identifier: observability_pipelines_syslog
weight: 1117
weight: 1118
- name: Environment Variables
url: observability_pipelines/environment_variables/
parent: observability_pipelines
Expand Down
39 changes: 39 additions & 0 deletions content/en/observability_pipelines/destinations/http_client.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
---
title: HTTP Client Destination
disable_toc: false
---

## Overview

Use Observability Pipelines' HTTP Client destination to send logs to an HTTP client, such as a logging platform or SIEM.

## Set up destination

Set up the New Relic destination and its environment variables when you [set up a pipeline][1]. The information below is configured in the pipelines UI.

1. Select your authorization strategy (**None**, **Basic**, or **Bearer**).
1. JSON is the only available encoder.
1. Optionally, toggle the switch to enable compression. If enabled:
1. GZIP is the only available compression algorithm.
1. Select the compression level you want to use.
1. Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.
- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509).
- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
- `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.

## Set the environment variables

{{% observability_pipelines/configure_existing_pipelines/destination_env_vars/http_client %}}

## How does the destination work

### Event batching

A batch of events is flushed when one of these parameters is met. See [event batching][2] for more information.

| Max Events | Max Bytes | Timeout (seconds) |
|----------------|-----------------|---------------------|
| 1,000 | 1,000,000 | 1 |

[1]: https://app.datadoghq.com/observability-pipelines
[2]: /observability_pipelines/destinations/#event-batching
3 changes: 3 additions & 0 deletions content/en/observability_pipelines/environment_variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,9 @@ Some Observability Pipelines components require setting up environment variables
### Elasticsearch
{{% observability_pipelines/configure_existing_pipelines/destination_env_vars/elasticsearch %}}

### HTTP Client
{{% observability_pipelines/configure_existing_pipelines/destination_env_vars/http_client %}}

### Microsoft Sentinel
{{% observability_pipelines/configure_existing_pipelines/destination_env_vars/microsoft_sentinel %}}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,11 @@ On the Worker installation page:

{{% observability_pipelines/configure_existing_pipelines/destination_env_vars/elasticsearch %}}

{{% /tab %}}
{{% tab "HTTP Client" %}}

{{% observability_pipelines/configure_existing_pipelines/destination_env_vars/http_client %}}

{{% /tab %}}
{{% tab "Microsoft Sentinel" %}}

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
- HTTP/S client URI endpoint
- Stored as the environment variable `DD_OP_DESTINATION_HTTP_CLIENT_URI`.
- If you are using basic authentication:
- HTTP/S endpoint authentication username and password.
- Stored as the environment variables: `DD_OP_DESTINATION_HTTP_CLIENT_USERNAME` and `DD_OP_DESTINATION_HTTP_CLIENT_PASSWORD`.
- If you are using bearer authentication:
- HTTP/S endpoint bearer token.
- Stored as the environment variable: `DD_OP_DESTINATION_HTTP_CLIENT_BEARER_TOKEN`.
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
| Elasticsearch | None | 10,000,000 | 1 |
| Google Chronicle | None | 1,000,000 | 15 |
| Google Cloud Storage (Datadog Log Archives)| None | 100,000,000 | 900 |
| HTTP Client | 1000 | 1,000,000 | 1 |
| Microsoft Sentinel | None | 10,000,000 | 1 |
| New Relic | 100 | 1,000,000 | 1 |
| OpenSearch | None | 10,000,000 | 1 |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,16 @@ To authenticate the Observability Pipelines Worker for Google Chronicle, contact
To set up the Worker's Google Chronicle destination:

1. Enter the customer ID for your Google Chronicle instance.
1. Enter the path to the credentials JSON file you downloaded earlier.
1. Optionally, enter the path to your credentials JSON file. If you configured Log Archives it's the credentials you downloaded [earlier](#create-a-service-account-to-allow-workers-to-write-to-the-bucket).
- You can also use the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to provide the credential path. If you're using [workload identity][10004] on Google Kubernetes Engine (GKE), the `GOOGLE_APPLICATION_CREDENTIALS` is provided for you.
- The Worker uses standard [Google authentication methods][10005].
1. Select **JSON** or **Raw** encoding in the dropdown menu.
1. Enter the log type. See [template syntax][10002] if you want to route logs to different log types based on specific fields in your logs.

**Note**: Logs sent to the Google Chronicle destination must have ingestion labels. For example, if the logs are from a A10 load balancer, it must have the ingestion label `A10_LOAD_BALANCER`. See Google Cloud's [Support log types with a default parser][10003] for a list of available log types and their respective ingestion labels.

[10001]: https://cloud.google.com/chronicle/docs/reference/ingestion-api#getting_api_authentication_credentials
[10002]: /observability_pipelines/destinations/#template-syntax
[10003]: https://cloud.google.com/chronicle/docs/ingestion/parser-list/supported-default-parsers#with-default-parser
[10003]: https://cloud.google.com/chronicle/docs/ingestion/parser-list/supported-default-parsers#with-default-parser
[10004]:https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity
[10005]: https://cloud.google.com/docs/authentication#auth-flowchart
Original file line number Diff line number Diff line change
@@ -1,12 +1,16 @@
<div class="alert alert-warning">The Google Cloud Storage destination only supports <a href = "https://cloud.google.com/storage/docs/access-control/lists">Access Control Lists</a>.</div>

1. Enter the name of the Google Cloud storage bucket you created earlier.
1. Enter the path to the credentials JSON file you downloaded [earlier](#create-a-service-account-to-allow-workers-to-write-to-the-bucket).
1. Enter the name of your Google Cloud storage bucket. If you configured Log Archives, it's the bucket you created earlier.
1. Optionally, enter the path to your credentials JSON file. If you configured Log Archives it's the credentials you downloaded [earlier](#create-a-service-account-to-allow-workers-to-write-to-the-bucket).
- You can also use the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to provide the credential path. If you're using [workload identity][10053] on Google Kubernetes Engine (GKE), the `GOOGLE_APPLICATION_CREDENTIALS` is provided for you.
- The Worker uses standard [Google authentication methods][10052].
1. Select the storage class for the created objects.
1. Select the access level of the created objects.
1. Optionally, enter in the prefix.
- Prefixes are useful for partitioning objects. For example, you can use a prefix as an object key to store objects under a particular directory. If using a prefix for this purpose, it must end in `/` to act as a directory path; a trailing `/` is not automatically added.
- See [template syntax][10051] if you want to route logs to different object keys based on specific fields in your logs.
1. Optionally, click **Add Header** to add metadata.

[10051]: /observability_pipelines/destinations/#template-syntax
[10051]: /observability_pipelines/destinations/#template-syntax
[10052]: https://cloud.google.com/docs/authentication#auth-flowchart
[10053]: https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity
Loading