Skip to content

Commit d0fc249

Browse files
Change first instances of data prepper to open search data prepper (#9091)
* Change first instances of Data Prepper to OpenSearch Data Prepper Signed-off-by: natebower <[email protected]> * Rest of changes Signed-off-by: natebower <[email protected]> * Change card Signed-off-by: Fanit Kolchina <[email protected]> * Change card Signed-off-by: Fanit Kolchina <[email protected]> * One more Signed-off-by: Fanit Kolchina <[email protected]> --------- Signed-off-by: natebower <[email protected]> Signed-off-by: Fanit Kolchina <[email protected]> Co-authored-by: Fanit Kolchina <[email protected]> Co-authored-by: kolchfa-aws <[email protected]>
1 parent d10f609 commit d0fc249

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

68 files changed

+97
-97
lines changed

_config.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -223,7 +223,7 @@ benchmark_collection:
223223
data_prepper_collection:
224224
collections:
225225
data-prepper:
226-
name: Data Prepper
226+
name: OpenSearch Data Prepper
227227
nav_fold: true
228228

229229
# Defaults
@@ -240,7 +240,7 @@ defaults:
240240
path: "_data-prepper"
241241
values:
242242
section: "data-prepper"
243-
section-name: "Data Prepper"
243+
section-name: "OpenSearch Data Prepper"
244244
-
245245
scope:
246246
path: "_clients"

_data-prepper/common-use-cases/anomaly-detection.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ nav_order: 5
77

88
# Anomaly detection
99

10-
You can use Data Prepper to train models and generate anomalies in near real time on time-series aggregated events. You can generate anomalies either on events generated within the pipeline or on events coming directly into the pipeline, like OpenTelemetry metrics. You can feed these tumbling window aggregated time-series events to the [`anomaly_detector` processor]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/anomaly-detector/), which trains a model and generates anomalies with a grade score. Then you can configure your pipeline to write the anomalies to a separate index to create document monitors and trigger fast alerting.
10+
You can use OpenSearch Data Prepper to train models and generate anomalies in near real time on time-series aggregated events. You can generate anomalies either on events generated within the pipeline or on events coming directly into the pipeline, like OpenTelemetry metrics. You can feed these tumbling window aggregated time-series events to the [`anomaly_detector` processor]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/anomaly-detector/), which trains a model and generates anomalies with a grade score. Then you can configure your pipeline to write the anomalies to a separate index to create document monitors and trigger fast alerting.
1111

1212
## Metrics from logs
1313

_data-prepper/common-use-cases/codec-processor-combinations.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ nav_order: 10
77

88
# Codec processor combinations
99

10-
At ingestion time, data received by the [`s3` source]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/s3/) can be parsed by [codecs]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/s3#codec). Codecs compresses and decompresses large data sets in a certain format before ingestion them through a Data Prepper pipeline [processor]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/processors/).
10+
At ingestion time, data received by the [`s3` source]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/s3/) can be parsed by [codecs]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/s3#codec). Codecs compresses and decompresses large data sets in a certain format before ingestion them through an OpenSearch Data Prepper pipeline [processor]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/processors/).
1111

1212
While most codecs can be used with most processors, the following codec processor combinations can make your pipeline more efficient when used with the following input types.
1313

_data-prepper/common-use-cases/common-use-cases.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,4 +9,4 @@ redirect_from:
99

1010
# Common use cases
1111

12-
You can use Data Prepper for several different purposes, including trace analytics, log analytics, Amazon S3 log analytics, and metrics ingestion.
12+
You can use OpenSearch Data Prepper for several different purposes, including trace analytics, log analytics, Amazon S3 log analytics, and metrics ingestion.

_data-prepper/common-use-cases/event-aggregation.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ nav_order: 25
77

88
# Event aggregation
99

10-
You can use Data Prepper to aggregate data from different events over a period of time. Aggregating events can help to reduce unnecessary log volume and manage use cases like multiline logs that are received as separate events. The [`aggregate` processor]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/aggregate/) is a stateful processor that groups events based on the values for a set of specified identification keys and performs a configurable action on each group.
10+
You can use OpenSearch Data Prepper to aggregate data from different events over a period of time. Aggregating events can help to reduce unnecessary log volume and manage use cases like multiline logs that are received as separate events. The [`aggregate` processor]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/aggregate/) is a stateful processor that groups events based on the values for a set of specified identification keys and performs a configurable action on each group.
1111

1212
The `aggregate` processor state is stored in memory. For example, in order to combine four events into one, the processor needs to retain pieces of the first three events. The state of an aggregate group of events is kept for a configurable amount of time. Depending on your logs, the aggregate action being used, and the number of memory options in the processor configuration, the aggregation could take place over a long period of time.
1313

_data-prepper/common-use-cases/log-analytics.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ nav_order: 30
77

88
# Log analytics
99

10-
Data Prepper is an extendable, configurable, and scalable solution for log ingestion into OpenSearch and Amazon OpenSearch Service. Data Prepper supports receiving logs from [Fluent Bit](https://fluentbit.io/) through the [HTTP Source](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/http-source/README.md) and processing those logs with a [Grok Processor](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/grok-processor/README.md) before ingesting them into OpenSearch through the [OpenSearch sink](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/opensearch/README.md).
10+
OpenSearch Data Prepper is an extendable, configurable, and scalable solution for log ingestion into OpenSearch and Amazon OpenSearch Service. Data Prepper supports receiving logs from [Fluent Bit](https://fluentbit.io/) through the [HTTP Source](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/http-source/README.md) and processing those logs with a [Grok Processor](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/grok-processor/README.md) before ingesting them into OpenSearch through the [OpenSearch sink](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/opensearch/README.md).
1111

1212
The following image shows all of the components used for log analytics with Fluent Bit, Data Prepper, and OpenSearch.
1313

_data-prepper/common-use-cases/log-enrichment.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ nav_order: 35
77

88
# Log enrichment
99

10-
You can perform different types of log enrichment with Data Prepper, including:
10+
You can perform different types of log enrichment with OpenSearch Data Prepper, including:
1111

1212
- Filtering.
1313
- Extracting key-value pairs from strings.

_data-prepper/common-use-cases/metrics-logs.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ nav_order: 15
77

88
# Deriving metrics from logs
99

10-
You can use Data Prepper to derive metrics from logs.
10+
You can use OpenSearch Data Prepper to derive metrics from logs.
1111

1212
The following example pipeline receives incoming logs using the [`http` source plugin]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/http-source) and the [`grok` processor]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/grok/). It then uses the [`aggregate` processor]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/aggregate/) to extract the metric bytes aggregated during a 30-second window and derives histograms from the results.
1313

_data-prepper/common-use-cases/metrics-traces.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ nav_order: 20
77

88
# Deriving metrics from traces
99

10-
You can use Data Prepper to derive metrics from OpenTelemetry traces. The following example pipeline receives incoming traces and extracts a metric called `durationInNanos`, aggregated over a tumbling window of 30 seconds. It then derives a histogram from the incoming traces.
10+
You can use OpenSearch Data Prepper to derive metrics from OpenTelemetry traces. The following example pipeline receives incoming traces and extracts a metric called `durationInNanos`, aggregated over a tumbling window of 30 seconds. It then derives a histogram from the incoming traces.
1111

1212
The pipeline contains the following pipelines:
1313

_data-prepper/common-use-cases/s3-logs.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ nav_order: 40
77

88
# S3 logs
99

10-
Data Prepper allows you to load logs from [Amazon Simple Storage Service](https://aws.amazon.com/s3/) (Amazon S3), including traditional logs, JSON documents, and CSV logs.
10+
OpenSearch Data Prepper allows you to load logs from [Amazon Simple Storage Service](https://aws.amazon.com/s3/) (Amazon S3), including traditional logs, JSON documents, and CSV logs.
1111

1212
## Architecture
1313

_data-prepper/common-use-cases/sampling.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ nav_order: 45
77

88
# Sampling
99

10-
Data Prepper provides the following sampling capabilities:
10+
OpenSearch Data Prepper provides the following sampling capabilities:
1111

1212
- Time sampling
1313
- Percentage sampling

_data-prepper/common-use-cases/text-processing.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ nav_order: 55
77

88
# Text processing
99

10-
Data Prepper provides text processing capabilities with the [`grok processor`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/grok/). The `grok` processor is based on the [`java-grok`](https://mvnrepository.com/artifact/io.krakens/java-grok) library and supports all compatible patterns. The `java-grok` library is built using the [`java.util.regex`](https://docs.oracle.com/javase/8/docs/api/java/util/regex/package-summary.html) regular expression library.
10+
OpenSearch Data Prepper provides text processing capabilities with the [`grok processor`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/grok/). The `grok` processor is based on the [`java-grok`](https://mvnrepository.com/artifact/io.krakens/java-grok) library and supports all compatible patterns. The `java-grok` library is built using the [`java.util.regex`](https://docs.oracle.com/javase/8/docs/api/java/util/regex/package-summary.html) regular expression library.
1111

1212
You can add custom patterns to your pipelines by using the `patterns_definitions` option. When debugging custom patterns, the [Grok Debugger](https://grokdebugger.com/) can be helpful.
1313

_data-prepper/common-use-cases/trace-analytics.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ nav_order: 60
77

88
# Trace analytics
99

10-
Trace analytics allows you to collect trace data and customize a pipeline that ingests and transforms the data for use in OpenSearch. The following provides an overview of the trace analytics workflow in Data Prepper, how to configure it, and how to visualize trace data.
10+
Trace analytics allows you to collect trace data and customize a pipeline that ingests and transforms the data for use in OpenSearch. The following provides an overview of the trace analytics workflow in OpenSearch Data Prepper, how to configure it, and how to visualize trace data.
1111

1212
## Introduction
1313

_data-prepper/getting-started.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,14 @@
11
---
22
layout: default
3-
title: Getting started
3+
title: Getting started with OpenSearch Data Prepper
44
nav_order: 5
55
redirect_from:
66
- /clients/data-prepper/get-started/
77
---
88

9-
# Getting started with Data Prepper
9+
# Getting started with OpenSearch Data Prepper
1010

11-
Data Prepper is an independent component, not an OpenSearch plugin, that converts data for use with OpenSearch. It's not bundled with the all-in-one OpenSearch installation packages.
11+
OpenSearch Data Prepper is an independent component, not an OpenSearch plugin, that converts data for use with OpenSearch. It's not bundled with the all-in-one OpenSearch installation packages.
1212

1313
If you are migrating from Open Distro Data Prepper, see [Migrating from Open Distro]({{site.url}}{{site.baseurl}}/data-prepper/migrate-open-distro/).
1414
{: .note}

_data-prepper/index.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
layout: default
3-
title: Data Prepper
3+
title: OpenSearch Data Prepper
44
nav_order: 1
55
has_children: false
66
has_toc: false
@@ -12,9 +12,9 @@ redirect_from:
1212
- /data-prepper/index/
1313
---
1414

15-
# Data Prepper
15+
# OpenSearch Data Prepper
1616

17-
Data Prepper is a server-side data collector capable of filtering, enriching, transforming, normalizing, and aggregating data for downstream analysis and visualization. Data Prepper is the preferred data ingestion tool for OpenSearch. It is recommended for most data ingestion use cases in OpenSearch and for processing large, complex datasets.
17+
OpenSearch Data Prepper is a server-side data collector capable of filtering, enriching, transforming, normalizing, and aggregating data for downstream analysis and visualization. Data Prepper is the preferred data ingestion tool for OpenSearch. It is recommended for most data ingestion use cases in OpenSearch and for processing large, complex datasets.
1818

1919
With Data Prepper you can build custom pipelines to improve the operational view of applications. Two common use cases for Data Prepper are trace analytics and log analytics. [Trace analytics]({{site.url}}{{site.baseurl}}/data-prepper/common-use-cases/trace-analytics/) can help you visualize event flows and identify performance problems. [Log analytics]({{site.url}}{{site.baseurl}}/data-prepper/common-use-cases/log-analytics/) equips you with tools to enhance your search capabilities, conduct comprehensive analysis, and gain insights into your applications' performance and behavior.
2020

@@ -74,6 +74,6 @@ In the given pipeline configuration, the `source` component reads string events
7474

7575
## Next steps
7676

77-
- [Get started with Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/getting-started/).
77+
- [Getting started with OpenSearch Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/getting-started/).
7878
- [Get familiar with Data Prepper pipelines]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/pipelines/).
7979
- [Explore common use cases]({{site.url}}{{site.baseurl}}/data-prepper/common-use-cases/common-use-cases/).

_data-prepper/managing-data-prepper/configuring-data-prepper.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,16 @@
11
---
22
layout: default
3-
title: Configuring Data Prepper
4-
parent: Managing Data Prepper
3+
title: Configuring OpenSearch Data Prepper
4+
parent: Managing OpenSearch Data Prepper
55
nav_order: 5
66
redirect_from:
77
- /clients/data-prepper/data-prepper-reference/
88
- /monitoring-plugins/trace/data-prepper-reference/
99
---
1010

11-
# Configuring Data Prepper
11+
# Configuring OpenSearch Data Prepper
1212

13-
You can customize your Data Prepper configuration by editing the `data-prepper-config.yaml` file in your Data Prepper installation. The following configuration options are independent from pipeline configuration options.
13+
You can customize your OpenSearch Data Prepper configuration by editing the `data-prepper-config.yaml` file in your Data Prepper installation. The following configuration options are independent from pipeline configuration options.
1414

1515

1616
## Data Prepper configuration

_data-prepper/managing-data-prepper/configuring-log4j.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,13 @@
11
---
22
layout: default
33
title: Configuring Log4j
4-
parent: Managing Data Prepper
4+
parent: Managing OpenSearch Data Prepper
55
nav_order: 20
66
---
77

88
# Configuring Log4j
99

10-
You can configure logging using Log4j in Data Prepper.
10+
You can configure logging using Log4j in OpenSearch Data Prepper.
1111

1212
## Logging
1313

_data-prepper/managing-data-prepper/core-apis.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,13 @@
11
---
22
layout: default
33
title: Core APIs
4-
parent: Managing Data Prepper
4+
parent: Managing OpenSearch Data Prepper
55
nav_order: 15
66
---
77

88
# Core APIs
99

10-
All Data Prepper instances expose a server with some control APIs. By default, this server runs on port 4900. Some plugins, especially source plugins, may expose other servers that run on different ports. Configurations for these plugins are independent of the core API. For example, to shut down Data Prepper, you can run the following curl request:
10+
All OpenSearch Data Prepper instances expose a server with some control APIs. By default, this server runs on port 4900. Some plugins, especially source plugins, may expose other servers that run on different ports. Configurations for these plugins are independent of the core API. For example, to shut down Data Prepper, you can run the following curl request:
1111

1212
```
1313
curl -X POST http://localhost:4900/shutdown

_data-prepper/managing-data-prepper/extensions/extensions.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,14 @@
11
---
22
layout: default
33
title: Extensions
4-
parent: Managing Data Prepper
4+
parent: Managing OpenSearch Data Prepper
55
has_children: true
66
nav_order: 18
77
---
88

99
# Extensions
1010

11-
Data Prepper extensions provide Data Prepper functionality outside of core Data Prepper pipeline components.
11+
OpenSearch Data Prepper extensions provide Data Prepper functionality outside of core Data Prepper pipeline components.
1212
Many extensions provide configuration options that give Data Prepper administrators greater flexibility over Data Prepper's functionality.
1313

1414
Extension configurations can be configured in the `data-prepper-config.yaml` file under the `extensions:` YAML block.

_data-prepper/managing-data-prepper/extensions/geoip-service.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,12 +3,12 @@ layout: default
33
title: geoip_service
44
nav_order: 5
55
parent: Extensions
6-
grand_parent: Managing Data Prepper
6+
grand_parent: Managing OpenSearch Data Prepper
77
---
88

99
# geoip_service
1010

11-
The `geoip_service` extension configures all [`geoip`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/geoip) processors in Data Prepper.
11+
The `geoip_service` extension configures all [`geoip`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/geoip) processors in OpenSearch Data Prepper.
1212

1313
## Usage
1414

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
---
22
layout: default
3-
title: Managing Data Prepper
3+
title: Managing OpenSearch Data Prepper
44
has_children: true
55
nav_order: 20
66
---
77

8-
# Managing Data Prepper
8+
# Managing OpenSearch Data Prepper
99

10-
You can perform administrator functions for Data Prepper, including system configuration, interacting with core APIs, Log4j configuration, and monitoring. You can set up peer forwarding to coordinate multiple Data Prepper nodes when using stateful aggregation.
10+
You can perform administrator functions for OpenSearch Data Prepper, including system configuration, interacting with core APIs, Log4j configuration, and monitoring. You can set up peer forwarding to coordinate multiple Data Prepper nodes when using stateful aggregation.

_data-prepper/managing-data-prepper/monitoring.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,13 @@
11
---
22
layout: default
33
title: Monitoring
4-
parent: Managing Data Prepper
4+
parent: Managing OpenSearch Data Prepper
55
nav_order: 25
66
---
77

8-
# Monitoring Data Prepper with metrics
8+
# Monitoring OpenSearch Data Prepper with metrics
99

10-
You can monitor Data Prepper with metrics using [Micrometer](https://micrometer.io/). There are two types of metrics: JVM/system metrics and plugin metrics. [Prometheus](https://prometheus.io/) is used as the default metrics backend.
10+
You can monitor OpenSearch Data Prepper with metrics using [Micrometer](https://micrometer.io/). There are two types of metrics: JVM/system metrics and plugin metrics. [Prometheus](https://prometheus.io/) is used as the default metrics backend.
1111

1212
## JVM and system metrics
1313

_data-prepper/managing-data-prepper/peer-forwarder.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,12 +2,12 @@
22
layout: default
33
title: Peer forwarder
44
nav_order: 12
5-
parent: Managing Data Prepper
5+
parent: Managing OpenSearch Data Prepper
66
---
77

88
# Peer forwarder
99

10-
Peer forwarder is an HTTP service that performs peer forwarding of an `event` between Data Prepper nodes for aggregation. This HTTP service uses a hash-ring approach to aggregate events and determine which Data Prepper node it should handle on a given trace before rerouting it to that node. Currently, peer forwarder is supported by the `aggregate`, `service_map_stateful`, and `otel_traces_raw` [processors]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/processors/).
10+
Peer forwarder is an HTTP service that performs peer forwarding of an `event` between OpenSearch Data Prepper nodes for aggregation. This HTTP service uses a hash-ring approach to aggregate events and determine which Data Prepper node it should handle on a given trace before rerouting it to that node. Currently, peer forwarder is supported by the `aggregate`, `service_map_stateful`, and `otel_traces_raw` [processors]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/processors/).
1111

1212
Peer Forwarder groups events based on the identification keys provided by the supported processors. For `service_map_stateful` and `otel_traces_raw`, the identification key is `traceId` by default and cannot be configured. The `aggregate` processor is configured using the `identification_keys` configuration option. From here, you can specify which keys to use for Peer Forwarder. See [Aggregate Processor page](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) for more information about identification keys.
1313

0 commit comments

Comments
 (0)