diff --git a/docs/modules/ROOT/nav.adoc b/docs/modules/ROOT/nav.adoc index 6ec1f5246c..d1f1bdaf1f 100644 --- a/docs/modules/ROOT/nav.adoc +++ b/docs/modules/ROOT/nav.adoc @@ -61,7 +61,7 @@ * xref:observation.adoc[Micrometer Observation] ** xref:observation/installing.adoc[Installing] ** xref:observation/introduction.adoc[Introduction] -** xref:observation/handler.adoc[Observation Handlers] +** xref:observation/components.adoc[Components] ** xref:observation/instrumenting.adoc[Instrumenting] ** xref:observation/testing.adoc[Testing] ** xref:observation/projects.adoc[Instrumented Projects] diff --git a/docs/modules/ROOT/pages/observation/handler.adoc b/docs/modules/ROOT/pages/observation/components.adoc similarity index 59% rename from docs/modules/ROOT/pages/observation/handler.adoc rename to docs/modules/ROOT/pages/observation/components.adoc index 3ee9f0b766..06185a1a71 100644 --- a/docs/modules/ROOT/pages/observation/handler.adoc +++ b/docs/modules/ROOT/pages/observation/components.adoc @@ -1,5 +1,69 @@ +[[micrometer-observation-components]] += Observation Components + +In this section we will describe main components related to Micrometer Observation. + +* <> +* <> +* <> +* <> +* <> + +*Micrometer Observation basic flow* + +`Observation` through `ObservationRegistry` gets created with a mutable `Observation.Context`. On each Micrometer Observation lifecycle action (e.g. `start()`) a corresponding `ObservationHandler` method is called (e.g. `onStart`) with the mutable `Observation.Context` as argument. + +// https://arthursonzogni.com/Diagon/#GraphDAG +// ObservationRegistry->Observation +// Context->Observation +// Observation->Handler +[source,subs=+attributes] +----- +┌───────────────────┐┌───────┐ +│ObservationRegistry││Context│ +└┬──────────────────┘└┬──────┘ +┌▽────────────────────▽┐ +│Observation │ +└┬─────────────────────┘ +┌▽──────┐ +│Handler│ +└───────┘ +----- + +*Micrometer Observation detailed flow* + +// https://arthursonzogni.com/Diagon/#GraphDAG +// ObservationRegistry->Observation +// Context->Observation +// ObservationConvention->Observation +// ObservationPredicate->Observation +// Observation->Handler +// Handler->ObservationFilter +[source,subs=+attributes] +----- +┌───────────────────┐┌───────┐┌─────────────────────┐┌────────────────────┐ +│ObservationRegistry││Context││ObservationConvention││ObservationPredicate│ +└┬──────────────────┘└┬──────┘└┬────────────────────┘└┬───────────────────┘ +┌▽────────────────────▽────────▽──────────────────────▽┐ +│Observation │ +└┬─────────────────────────────────────────────────────┘ +┌▽──────┐ +│Handler│ +└┬──────┘ +┌▽────────────────┐ +│ObservationFilter│ +└─────────────────┘ +----- + +`Observation` through `ObservationRegistry` gets created with a mutable `Observation.Context`. To allow name and key-value customization, an `ObservationConvention` can be used instead of direct name setting. List of `ObservationPredicate` is run to verify if an `Observation` should be created instead of a no-op version. On each Micrometer Observation lifecycle action (e.g. `start()`) a corresponding `ObservationHandler` method is called (e.g. `onStart`) with the mutable `Observation.Context` as argument. On `Observation` stop, before calling the `ObservationHandler` `onStop` methods, list of `ObservationFilter` is called to optionally further modify the `Observation.Context`. + +[[micrometer-observation-context]] +== Observation.Context + +To pass information between the instrumented code and the handler (or between handler methods, such as `onStart` and `onStop`), you can use an `Observation.Context`. An `Observation.Context` is a `Map`-like container that can store values for you while your handler can access the data inside the context. + [[micrometer-observation-handler]] -= Observation Handler +== Observation Handler A popular way to record Observations is storing the start state in a `Timer.Sample` instance and stopping it when the event has ended. Recording such measurements could look like this: @@ -19,13 +83,8 @@ include::{include-java}/observation/ObservationHandlerTests.java[tags=observatio Starting with Micrometer 1.10, you can register "handlers" (`ObservationHandler` instances) that are notified about the lifecycle event of an observation (for example, you can run custom code when an observation is started or stopped). Using this feature lets you add tracing capabilities to your existing metrics instrumentation (see: `DefaultTracingObservationHandler`). The implementation of these handlers does not need to be tracing related. It is completely up to you how you are going to implement them (for example, you can add logging capabilities). -[[micrometer-observation-context]] -== Observation.Context - -To pass information between the instrumented code and the handler (or between handler methods, such as `onStart` and `onStop`), you can use an `Observation.Context`. An `Observation.Context` is a `Map`-like container that can store values for you while your handler can access the data inside the context. - [[micrometer-observation-handler-example]] -== ObservationHandler Example +=== ObservationHandler Example Based on this, we can implement a simple handler that lets the users know about its invocations by printing them out to `stdout`: diff --git a/docs/modules/ROOT/pages/observation/instrumenting.adoc b/docs/modules/ROOT/pages/observation/instrumenting.adoc index 83e919fe56..1ae78f8234 100644 --- a/docs/modules/ROOT/pages/observation/instrumenting.adoc +++ b/docs/modules/ROOT/pages/observation/instrumenting.adoc @@ -1,6 +1,8 @@ [[micrometer-observation-handler]] = Instrumentation +IMPORTANT: This section of the documentation is targeted to users who want to add instrumentation to their codebase. + In this section, we see some common examples of reusing existing Micrometer and Micrometer Tracing handlers and context types to do instrumentation. IMPORTANT: Before you decide to instrument a project yourself, double-check whether that it has not already been instrumented! @@ -14,68 +16,198 @@ To better convey how you can do instrumentation, we need to distinguish two conc *Creation of Observations* - We want to wrap an operation in an Observation to get measurements. We need to know if there previously has been a parent Observation to maintain the parent-child relationship of Observations. -[[instrumentation_of_thread_switching_components]] -== Instrumentation of Thread Switching Components - -We might want to create an Observation around a `Runnable` or `Callable` that we're submitting through an `Executor`. For that to work, we need to know if there was an Observation in the parent thread that the new thread should continue or for which a child Observation should be created. - -Consider the following example: - -[source,java] ------ -include::{include-java}/observation/ObservationInstrumentingTests.java[tags=executor,indent=0] - -include::{include-java}/observation/ObservationInstrumentingTests.java[tags=thread_switching,indent=0] ------ - -[[instrumentation_of_reactive_libraries]] -== Instrumentation of Reactive Libraries - -In this section, we discuss how to wrap Reactive libraries in Observations and how to use Reactor Context to safely propagate Observations between threads. - -[[instrumentation_of_reactive_libraries_after_reactor_3_5_3]] -=== For Reactor 3.5.3 and After - -In the Reactor 3.5.3 release (through this https://github.com/reactor/reactor-core/pull/3335[PR]), an option to turn on automated context propagation was added. To use this, ensure that you use the following projects at minimum in the following versions: - -- Reactor https://github.com/reactor/reactor-core/releases/tag/v3.5.7[3.5.7] -- Micrometer Context-Propagation https://github.com/micrometer-metrics/context-propagation/releases/tag/v1.0.3[1.0.3] -- Micrometer https://github.com/micrometer-metrics/micrometer/releases/tag/v1.10.8[1.10.8] -- Micrometer Tracing https://github.com/micrometer-metrics/tracing/releases/tag/v1.0.7[1.0.7] - -To use the feature, call the new Reactor's Hook method (for example, in your `public static void main` method), like this: - -[source,java] ------ -include::{include-java}/observation/ObservationInstrumentingTests.java[tags=reactor_hook,indent=0] ------ - -This automatically wraps Reactor's internal mechanisms to propagate context between operators, threads, and so on. Usage of `tap` and `handle` or the Context Propagation API is not required. - -Consider the following example: - -[source,java] ------ -include::{include-java}/observation/ObservationInstrumentingTests.java[tags=reactor_with_hook,indent=0] ------ - -If the performance of this approach is not satisfactory, check whether disabling the hook and explicitly using `handle` or `tap` operators improves the performance. - -[[instrumentation_of_reactive_libraries_before_reactor_3_5_3]] -=== Before Reactor 3.5.3 - -The preferred way of propagating elements through the Flux by using Reactor is not through `ThreadLocal` instances but through Reactor Context. Reactor, however, gives you two operators: `tap()` and `handle()`. With these two operators, if the https://docs.micrometer.io/context-propagation/reference/[Micrometer Context Propagation] library is on the classpath, it sets the thread local values for you. - -Consider the following example: - -[source,java] ------ -include::{include-java}/observation/ObservationInstrumentingTests.java[tags=reactor,indent=0] ------ - [[instrumentation_of_http_communication]] == Instrumentation of HTTP Communication +In this section you can find how to instrument libraries that do HTTP communication. + +[[instrumentation_of_http_communication_client]] +=== Instrumentation of HTTP Client Communication + +*Basic explanation of HTTP client side instrumentation* + +// https://arthursonzogni.com/Diagon/#Sequence +// RequestReplySenderContext -> Carrier: Wrap +// ObservationRegistry -> Observation: Create +// RequestReplySenderContext -> Observation: Create +// ObservationConvention -> Observation: Create +// Observation -> Code to Instrument: Wrap in Scope +[source,subs=+attributes] +----- +┌─────────────────────────┐┌───────┐┌───────────────────┐┌───────────┐┌─────────────────────┐┌──────────────────┐ +│RequestReplySenderContext││Carrier││ObservationRegistry││Observation││ObservationConvention││Code to Instrument│ +└────────────┬────────────┘└───┬───┘└─────────┬─────────┘└─────┬─────┘└──────────┬──────────┘└────────┬─────────┘ + │ │ │ │ │ │ + │ Wrap │ │ │ │ │ + │────────────────>│ │ │ │ │ + │ │ │ │ │ │ + │ │ │ Create │ │ │ + │ │ │───────────────>│ │ │ + │ │ │ │ │ │ + │ │ Create │ │ │ │ + │────────────────────────────────────────────────>│ │ │ + │ │ │ │ │ │ + │ │ │ │ Create │ │ + │ │ │ │<────────────────│ │ + │ │ │ │ │ │ + │ │ │ │ Wrap in Scope │ + │ │ │ │─────────────────────────────────────>│ +┌────────────┴────────────┐┌───┴───┐┌─────────┴─────────┐┌─────┴─────┐┌──────────┴──────────┐┌────────┴─────────┐ +│RequestReplySenderContext││Carrier││ObservationRegistry││Observation││ObservationConvention││Code to Instrument│ +└─────────────────────────┘└───────┘└───────────────────┘└───────────┘└─────────────────────┘└──────────────────┘ +----- + +. Create a `RequestReplySenderContext` that wraps a carrier (e.g. `HttpRequest`) +.. In its constructor explain how to enrich the headers (e.g. `(key, value) -> httpRequest.header(key, value)`) +.. Set the carrier on the `RequestReplySenderContext` +. Create an `Observation`, optionally using the `ObservationConvention` with the sender context +.. On `Observation` start, assuming that there is a proper registered handler through the `ObservationRegistry`, propagation will happen (e.g. carrier will be enriched with proper headers) +. Wrap the code to instrument (e.g. sending of an HTTP request) in scope (e.g. through the `observe` or `scoped` method) + +*More detailed explanation of HTTP client side instrumentation* + +// https://arthursonzogni.com/Diagon/#Sequence +// RequestReplySenderContext -> Carrier: Wrap +// ObservationConvention -> Observation: Create +// RequestReplySenderContext -> Observation: Create +// Observation -> ObservationHandler: Start +// ObservationHandler -> Propagator: Instrument Carrier +// Propagator -> Carrier: Instrument Carrier +// Observation -> Code to Instrument: Wrap in Scope +[source,subs=+attributes] +----- +┌─────────────────────────┐┌───────┐┌─────────────────────┐┌───────────┐┌──────────────────┐ ┌──────────┐┌──────────────────┐ +│RequestReplySenderContext││Carrier││ObservationConvention││Observation││ObservationHandler│ │Propagator││Code to Instrument│ +└────────────┬────────────┘└───┬───┘└──────────┬──────────┘└─────┬─────┘└────────┬─────────┘ └────┬─────┘└────────┬─────────┘ + │ │ │ │ │ │ │ + │ Wrap │ │ │ │ │ │ + │────────────────>│ │ │ │ │ │ + │ │ │ │ │ │ │ + │ │ │ Create │ │ │ │ + │ │ │────────────────>│ │ │ │ + │ │ │ │ │ │ │ + │ │ Create │ │ │ │ │ + │──────────────────────────────────────────────────>│ │ │ │ + │ │ │ │ │ │ │ + │ │ │ │ Start │ │ │ + │ │ │ │──────────────>│ │ │ + │ │ │ │ │ │ │ + │ │ │ │ │Instrument Carrier│ │ + │ │ │ │ │─────────────────>│ │ + │ │ │ │ │ │ │ + │ │ │ Instrument Carrier │ │ │ + │ │<───────────────────────────────────────────────────────────────────│ │ + │ │ │ │ │ │ │ + │ │ │ │ │ Wrap in Scope │ │ + │ │ │ │─────────────────────────────────────────────────>│ +┌────────────┴────────────┐┌───┴───┐┌──────────┴──────────┐┌─────┴─────┐┌────────┴─────────┐ ┌────┴─────┐┌────────┴─────────┐ +│RequestReplySenderContext││Carrier││ObservationConvention││Observation││ObservationHandler│ │Propagator││Code to Instrument│ +└─────────────────────────┘└───────┘└─────────────────────┘└───────────┘└──────────────────┘ └──────────┘└──────────────────┘ +----- + +. In the `ObservationRegistry` register a handler that will propagate context (e.g. `PropagatingSenderTracingObservationHandler` from Micrometer Tracing) +. Create a `RequestReplySenderContext` that wraps a carrier (e.g. `HttpRequest`) +.. In its constructor explain how to enrich the headers (e.g. `(carrier, key, value) -> carrier.header(key, value)`) +.. Set the carrier on the `RequestReplySenderContext` +. Create an `Observation`, optionally using the `ObservationConvention` with the sender context +.. On `Observation` start the registered handler will instrument the carrier +. Wrap the code to instrument (e.g. sending of an HTTP request) in scope (e.g. through the `observe` or `scoped` method) + +[[instrumentation_of_http_communication_server]] +=== Instrumentation of HTTP Server Communication + +*Basic explanation of HTTP server side instrumentation* + +// https://arthursonzogni.com/Diagon/#Sequence +// RequestReplyReceiverContext -> Carrier: Wrap +// ObservationRegistry -> Observation: Create +// RequestReplyReceiverContext -> Observation: Create +// ObservationConvention -> Observation: Create +// Observation -> Code to Instrument: Wrap in Scope +[source,subs=+attributes] +----- +┌───────────────────────────┐┌───────┐┌───────────────────┐┌───────────┐┌─────────────────────┐┌──────────────────┐ +│RequestReplyReceiverContext││Carrier││ObservationRegistry││Observation││ObservationConvention││Code to Instrument│ +└─────────────┬─────────────┘└───┬───┘└─────────┬─────────┘└─────┬─────┘└──────────┬──────────┘└────────┬─────────┘ + │ │ │ │ │ │ + │ Wrap │ │ │ │ │ + │─────────────────>│ │ │ │ │ + │ │ │ │ │ │ + │ │ │ Create │ │ │ + │ │ │───────────────>│ │ │ + │ │ │ │ │ │ + │ │ Create │ │ │ │ + │─────────────────────────────────────────────────>│ │ │ + │ │ │ │ │ │ + │ │ │ │ Create │ │ + │ │ │ │<────────────────│ │ + │ │ │ │ │ │ + │ │ │ │ Wrap in Scope │ + │ │ │ │─────────────────────────────────────>│ +┌─────────────┴─────────────┐┌───┴───┐┌─────────┴─────────┐┌─────┴─────┐┌──────────┴──────────┐┌────────┴─────────┐ +│RequestReplyReceiverContext││Carrier││ObservationRegistry││Observation││ObservationConvention││Code to Instrument│ +└───────────────────────────┘└───────┘└───────────────────┘└───────────┘└─────────────────────┘└──────────────────┘ +----- + +. Create a `RequestReplyReceiverContext` that wraps a carrier (e.g. `HttpRequest`) +.. In its constructor explain how to retrieve the header values (e.g. `(carrier, key) -> carrier.header(key)`) +.. Set the carrier on the `RequestReplyReceiverContext` +. Create an `Observation`, optionally using the `ObservationConvention` with the sender context +.. On `Observation` start, assuming that there is a proper registered handler through the `ObservationRegistry`, propagation will happen (e.g. tracing information will be retrieved from the headers) +. Wrap the code to instrument (e.g. processing of an HTTP request) in scope (e.g. through the `observe` or `scoped` method) + +*More detailed explanation of HTTP client side instrumentation* + +// https://arthursonzogni.com/Diagon/#Sequence +// RequestReplyReceiverContext -> Carrier: Wrap +// ObservationConvention -> Observation: Create +// RequestReplyReceiverContext -> Observation: Create +// Observation -> ObservationHandler: Start +// ObservationHandler -> Propagator: Extract Carrier +// Propagator -> Carrier: Extract Carrier +// Observation -> Code to Instrument: Wrap in Scope +[source,subs=+attributes] +----- +┌───────────────────────────┐┌───────┐┌─────────────────────┐┌───────────┐┌──────────────────┐┌──────────┐┌──────────────────┐ +│RequestReplyReceiverContext││Carrier││ObservationConvention││Observation││ObservationHandler││Propagator││Code to Instrument│ +└─────────────┬─────────────┘└───┬───┘└──────────┬──────────┘└─────┬─────┘└────────┬─────────┘└────┬─────┘└────────┬─────────┘ + │ │ │ │ │ │ │ + │ Wrap │ │ │ │ │ │ + │─────────────────>│ │ │ │ │ │ + │ │ │ │ │ │ │ + │ │ │ Create │ │ │ │ + │ │ │────────────────>│ │ │ │ + │ │ │ │ │ │ │ + │ │ Create │ │ │ │ │ + │───────────────────────────────────────────────────>│ │ │ │ + │ │ │ │ │ │ │ + │ │ │ │ Start │ │ │ + │ │ │ │──────────────>│ │ │ + │ │ │ │ │ │ │ + │ │ │ │ │Extract Carrier│ │ + │ │ │ │ │──────────────>│ │ + │ │ │ │ │ │ │ + │ │ │ Extract Carrier │ │ │ + │ │<────────────────────────────────────────────────────────────────│ │ + │ │ │ │ │ │ │ + │ │ │ │ │ Wrap in Scope │ │ + │ │ │ │──────────────────────────────────────────────>│ +┌─────────────┴─────────────┐┌───┴───┐┌──────────┴──────────┐┌─────┴─────┐┌────────┴─────────┐┌────┴─────┐┌────────┴─────────┐ +│RequestReplyReceiverContext││Carrier││ObservationConvention││Observation││ObservationHandler││Propagator││Code to Instrument│ +└───────────────────────────┘└───────┘└─────────────────────┘└───────────┘└──────────────────┘└──────────┘└──────────────────┘ +----- + +. In the `ObservationRegistry` register a handler that will propagate context (e.g. `PropagatingReceiverTracingObservationHandler` from Micrometer Tracing) +. Create a `RequestReplyReceiverContext` that wraps a carrier (e.g. `HttpRequest`) +.. In its constructor explain how to retrieve the header values (e.g. `(carrier, key) -> carrier.header(key)`) +.. Set the carrier on the `RequestReplyReceiverContext` +. Create an `Observation`, optionally using the `ObservationConvention` with the sender context +.. On `Observation` start, propagation will happen (e.g. tracing information will be retrieved from the headers) +. Wrap the code to instrument (e.g. processing of an HTTP request) in scope (e.g. through the `observe` or `scoped` method) + +[[instrumentation_of_http_communication_example]] +=== Instrumentation of HTTP Communication Example + To instrument an HTTP-based communication, we need to use the `RequestReplySenderContext` and `RequestReplyReceiverContext` for the client and server side, respectively. As an example for the client side, we use a handler that instruments the HTTP request by adding a `foo:bar` header (if you have Micrometer Tracing on the classpath, you could reuse the `PropagatingSenderTracingObservationHandler` and `PropagatingReceiverTracingObservationHandler` to propagate tracing context over the wire). Let's consider an example of such a handler: @@ -106,8 +238,200 @@ Consider the following HTTP server side instrumentation that reuses the handler: include::{include-java}/observation/ObservationInstrumentingTests.java[tags=http_server,indent=0] ----- +[[instrumentation_of_messaging_communication]] == Instrumentation of Messaging Communication +In this section you can find how to instrument libraries that do fire-and-forget like communication. + +[[instrumentation_of_messaging_communication_producer]] +=== Instrumentation of Messaging Producer Side + +*Basic explanation of messaging producer side instrumentation* + +// https://arthursonzogni.com/Diagon/#Sequence +// SenderContext -> Carrier: Wrap +// ObservationRegistry -> Observation: Create +// SenderContext -> Observation: Create +// ObservationConvention -> Observation: Create +// Observation -> Code to Instrument: Wrap in Scope +[source,subs=+attributes] +----- +┌─────────────┐┌───────┐┌───────────────────┐┌───────────┐┌─────────────────────┐┌──────────────────┐ +│SenderContext││Carrier││ObservationRegistry││Observation││ObservationConvention││Code to Instrument│ +└──────┬──────┘└───┬───┘└─────────┬─────────┘└─────┬─────┘└──────────┬──────────┘└────────┬─────────┘ + │ │ │ │ │ │ + │ Wrap │ │ │ │ │ + │──────────>│ │ │ │ │ + │ │ │ │ │ │ + │ │ │ Create │ │ │ + │ │ │───────────────>│ │ │ + │ │ │ │ │ │ + │ │ Create │ │ │ │ + │──────────────────────────────────────────>│ │ │ + │ │ │ │ │ │ + │ │ │ │ Create │ │ + │ │ │ │<────────────────│ │ + │ │ │ │ │ │ + │ │ │ │ Wrap in Scope │ + │ │ │ │─────────────────────────────────────>│ +┌──────┴──────┐┌───┴───┐┌─────────┴─────────┐┌─────┴─────┐┌──────────┴──────────┐┌────────┴─────────┐ +│SenderContext││Carrier││ObservationRegistry││Observation││ObservationConvention││Code to Instrument│ +└─────────────┘└───────┘└───────────────────┘└───────────┘└─────────────────────┘└──────────────────┘ +----- + +. Create a `SenderContext` that wraps a carrier (e.g. `AmqpMessage`) +.. In its constructor explain how to enrich the headers (e.g. `(key, value) -> amqpMessage.header(key, value)`) +.. Set the carrier on the `SenderContext` +. Create an `Observation`, optionally using the `ObservationConvention` with the sender context +.. On `Observation` start, assuming that there is a proper registered handler through the `ObservationRegistry`, propagation will happen (e.g. carrier will be enriched with proper headers) +. Wrap the code to instrument (e.g. sending of an AMQP message) in scope (e.g. through the `observe` or `scoped` method) + +*More detailed explanation of messaging producer side instrumentation* + +// https://arthursonzogni.com/Diagon/#Sequence +// SenderContext -> Carrier: Wrap +// ObservationConvention -> Observation: Create +// SenderContext -> Observation: Create +// Observation -> ObservationHandler: Start +// ObservationHandler -> Propagator: Instrument Carrier +// Propagator -> Carrier: Instrument Carrier +// Observation -> Code to Instrument: Wrap in Scope +[source,subs=+attributes] +----- +┌─────────────┐┌───────┐┌─────────────────────┐┌───────────┐┌──────────────────┐ ┌──────────┐┌──────────────────┐ +│SenderContext││Carrier││ObservationConvention││Observation││ObservationHandler│ │Propagator││Code to Instrument│ +└──────┬──────┘└───┬───┘└──────────┬──────────┘└─────┬─────┘└────────┬─────────┘ └────┬─────┘└────────┬─────────┘ + │ │ │ │ │ │ │ + │ Wrap │ │ │ │ │ │ + │──────────>│ │ │ │ │ │ + │ │ │ │ │ │ │ + │ │ │ Create │ │ │ │ + │ │ │────────────────>│ │ │ │ + │ │ │ │ │ │ │ + │ │ Create │ │ │ │ │ + │────────────────────────────────────────────>│ │ │ │ + │ │ │ │ │ │ │ + │ │ │ │ Start │ │ │ + │ │ │ │──────────────>│ │ │ + │ │ │ │ │ │ │ + │ │ │ │ │Instrument Carrier│ │ + │ │ │ │ │─────────────────>│ │ + │ │ │ │ │ │ │ + │ │ │ Instrument Carrier │ │ │ + │ │<───────────────────────────────────────────────────────────────────│ │ + │ │ │ │ │ │ │ + │ │ │ │ │ Wrap in Scope │ │ + │ │ │ │─────────────────────────────────────────────────>│ +┌──────┴──────┐┌───┴───┐┌──────────┴──────────┐┌─────┴─────┐┌────────┴─────────┐ ┌────┴─────┐┌────────┴─────────┐ +│SenderContext││Carrier││ObservationConvention││Observation││ObservationHandler│ │Propagator││Code to Instrument│ +└─────────────┘└───────┘└─────────────────────┘└───────────┘└──────────────────┘ └──────────┘└──────────────────┘ +----- + +. In the `ObservationRegistry` register a handler that will propagate context (e.g. `PropagatingReceiverTracingObservationHandler` from Micrometer Tracing) +. Create a `SenderContext` that wraps a carrier (e.g. `AmqpMessage`) +.. In its constructor explain how to enrich the headers (e.g. `(key, value) -> amqpMessage.header(key, value)`) +.. Set the carrier on the `SenderContext` +. Create an `Observation`, optionally using the `ObservationConvention` with the sender context +.. On `Observation` start, propagation will happen (e.g. carrier will be enriched with proper headers) +. Wrap the code to instrument (e.g. sending of an AMQP message) in scope (e.g. through the `observe` or `scoped` method) + +[[instrumentation_of_messaging_communication_consumer]] +=== Instrumentation of Messaging Consumer Side Communication + +*Basic explanation of messaging consumer side instrumentation* + +// https://arthursonzogni.com/Diagon/#Sequence +// ReceiverContext -> Carrier: Wrap +// ObservationRegistry -> Observation: Create +// ReceiverContext -> Observation: Create +// ObservationConvention -> Observation: Create +// Observation -> Code to Instrument: Wrap in Scope +[source,subs=+attributes] +----- +┌───────────────┐┌───────┐┌───────────────────┐┌───────────┐┌─────────────────────┐┌──────────────────┐ +│ReceiverContext││Carrier││ObservationRegistry││Observation││ObservationConvention││Code to Instrument│ +└───────┬───────┘└───┬───┘└─────────┬─────────┘└─────┬─────┘└──────────┬──────────┘└────────┬─────────┘ + │ │ │ │ │ │ + │ Wrap │ │ │ │ │ + │───────────>│ │ │ │ │ + │ │ │ │ │ │ + │ │ │ Create │ │ │ + │ │ │───────────────>│ │ │ + │ │ │ │ │ │ + │ │ Create │ │ │ │ + │───────────────────────────────────────────>│ │ │ + │ │ │ │ │ │ + │ │ │ │ Create │ │ + │ │ │ │<────────────────│ │ + │ │ │ │ │ │ + │ │ │ │ Wrap in Scope │ + │ │ │ │─────────────────────────────────────>│ +┌───────┴───────┐┌───┴───┐┌─────────┴─────────┐┌─────┴─────┐┌──────────┴──────────┐┌────────┴─────────┐ +│ReceiverContext││Carrier││ObservationRegistry││Observation││ObservationConvention││Code to Instrument│ +└───────────────┘└───────┘└───────────────────┘└───────────┘└─────────────────────┘└──────────────────┘ +----- + +. Create a `ReceiverContext` that wraps a carrier (e.g. `AmqpMessage`) +.. In its constructor explain how to retrieve the header values (e.g. `(carrier, key) -> carrier.header(key)`) +.. Set the carrier on the `ReceiverContext` +. Create an `Observation`, optionally using the `ObservationConvention` with the sender context +.. On `Observation` start, assuming that there is a proper registered handler through the `ObservationRegistry`, propagation will happen (e.g. tracing information will be retrieved from the headers) +. Wrap the code to instrument (e.g. processing of an HTTP request) in scope (e.g. through the `observe` or `scoped` method) +.. Some libraries (e.g. RabbitMQ) you might not have a handle on user's code, and you may require the user to allow starting a consumer side Observation and opening its scope by the framework (putting values in thread local) with the requirement of manually closing both the scope and stopping the Observation later by the user in their code! + +*More detailed explanation of messaging consumer side instrumentation* + +// https://arthursonzogni.com/Diagon/#Sequence +// ReceiverContext -> Carrier: Wrap +// ObservationConvention -> Observation: Create +// ReceiverContext -> Observation: Create +// Observation -> ObservationHandler: Start +// ObservationHandler -> Propagator: Extract Carrier +// Propagator -> Carrier: Extract Carrier +// Observation -> Code to Instrument: Wrap in Scope +[source,subs=+attributes] +----- +┌───────────────┐┌───────┐┌─────────────────────┐┌───────────┐┌──────────────────┐┌──────────┐┌──────────────────┐ +│ReceiverContext││Carrier││ObservationConvention││Observation││ObservationHandler││Propagator││Code to Instrument│ +└───────┬───────┘└───┬───┘└──────────┬──────────┘└─────┬─────┘└────────┬─────────┘└────┬─────┘└────────┬─────────┘ + │ │ │ │ │ │ │ + │ Wrap │ │ │ │ │ │ + │───────────>│ │ │ │ │ │ + │ │ │ │ │ │ │ + │ │ │ Create │ │ │ │ + │ │ │────────────────>│ │ │ │ + │ │ │ │ │ │ │ + │ │ Create │ │ │ │ │ + │─────────────────────────────────────────────>│ │ │ │ + │ │ │ │ │ │ │ + │ │ │ │ Start │ │ │ + │ │ │ │──────────────>│ │ │ + │ │ │ │ │ │ │ + │ │ │ │ │Extract Carrier│ │ + │ │ │ │ │──────────────>│ │ + │ │ │ │ │ │ │ + │ │ │ Extract Carrier │ │ │ + │ │<────────────────────────────────────────────────────────────────│ │ + │ │ │ │ │ │ │ + │ │ │ │ │ Wrap in Scope │ │ + │ │ │ │──────────────────────────────────────────────>│ +┌───────┴───────┐┌───┴───┐┌──────────┴──────────┐┌─────┴─────┐┌────────┴─────────┐┌────┴─────┐┌────────┴─────────┐ +│ReceiverContext││Carrier││ObservationConvention││Observation││ObservationHandler││Propagator││Code to Instrument│ +└───────────────┘└───────┘└─────────────────────┘└───────────┘└──────────────────┘└──────────┘└──────────────────┘ +----- + +. In the `ObservationRegistry` register a handler that will propagate context (e.g. `PropagatingReceiverTracingObservationHandler` from Micrometer Tracing) +. Create a `ReceiverContext` that wraps a carrier (e.g. `AmqpMessage`) +.. In its constructor explain how to retrieve the header values (e.g. `(carrier, key) -> carrier.header(key)`) +.. Set the carrier on the `ReceiverContext` +. Create an `Observation`, optionally using the `ObservationConvention` with the sender context +.. On `Observation` start, propagation will happen (e.g. tracing information will be retrieved from the headers) +. Wrap the code to instrument (e.g. processing of an HTTP request) in scope (e.g. through the `observe` or `scoped` method) +.. Some libraries (e.g. RabbitMQ) you might not have a handle on user's code, and you may require the user to allow starting a consumer side Observation and opening its scope by the framework (putting values in thread local) with the requirement of manually closing both the scope and stopping the Observation later by the user in their code! + +[[instrumentation_of_messaging_communication_example]] +=== Instrumentation of Messaging Communication Example + To instrument messaging-based communication we need to use the `SenderContext` and `ReceiverContext` for the producer and consumer side respectively. In this section we will create a simple instrumentation for Apache Kafka. @@ -180,3 +504,63 @@ Let's look at the assertions after having sent and received a message. We should ----- include::{include-java}/observation/messaging/ObservationMessagingIntegrationTest.java[tags=test_assertions,indent=0] ----- + + +[[instrumentation_of_thread_switching_components]] +== Instrumentation of Thread Switching Components + +We might want to create an Observation around a `Runnable` or `Callable` that we're submitting through an `Executor`. For that to work, we need to know if there was an Observation in the parent thread that the new thread should continue or for which a child Observation should be created. + +Consider the following example: + +[source,java] +----- +include::{include-java}/observation/ObservationInstrumentingTests.java[tags=executor,indent=0] + +include::{include-java}/observation/ObservationInstrumentingTests.java[tags=thread_switching,indent=0] +----- + +[[instrumentation_of_reactive_libraries]] +== Instrumentation of Reactive Libraries + +In this section, we discuss how to wrap Reactive libraries in Observations and how to use Reactor Context to safely propagate Observations between threads. + +[[instrumentation_of_reactive_libraries_after_reactor_3_5_3]] +=== For Reactor 3.5.3 and After + +In the Reactor 3.5.3 release (through this https://github.com/reactor/reactor-core/pull/3335[PR]), an option to turn on automated context propagation was added. To use this, ensure that you use the following projects at minimum in the following versions: + +- Reactor https://github.com/reactor/reactor-core/releases/tag/v3.5.7[3.5.7] +- Micrometer Context-Propagation https://github.com/micrometer-metrics/context-propagation/releases/tag/v1.0.3[1.0.3] +- Micrometer https://github.com/micrometer-metrics/micrometer/releases/tag/v1.10.8[1.10.8] +- Micrometer Tracing https://github.com/micrometer-metrics/tracing/releases/tag/v1.0.7[1.0.7] + +To use the feature, call the new Reactor's Hook method (for example, in your `public static void main` method), like this: + +[source,java] +----- +include::{include-java}/observation/ObservationInstrumentingTests.java[tags=reactor_hook,indent=0] +----- + +This automatically wraps Reactor's internal mechanisms to propagate context between operators, threads, and so on. Usage of `tap` and `handle` or the Context Propagation API is not required. + +Consider the following example: + +[source,java] +----- +include::{include-java}/observation/ObservationInstrumentingTests.java[tags=reactor_with_hook,indent=0] +----- + +If the performance of this approach is not satisfactory, check whether disabling the hook and explicitly using `handle` or `tap` operators improves the performance. + +[[instrumentation_of_reactive_libraries_before_reactor_3_5_3]] +=== Before Reactor 3.5.3 + +The preferred way of propagating elements through the Flux by using Reactor is not through `ThreadLocal` instances but through Reactor Context. Reactor, however, gives you two operators: `tap()` and `handle()`. With these two operators, if the https://docs.micrometer.io/context-propagation/reference/[Micrometer Context Propagation] library is on the classpath, it sets the thread local values for you. + +Consider the following example: + +[source,java] +----- +include::{include-java}/observation/ObservationInstrumentingTests.java[tags=reactor,indent=0] +----- diff --git a/docs/modules/ROOT/pages/observation/introduction.adoc b/docs/modules/ROOT/pages/observation/introduction.adoc index 16cc451b17..c37ca16989 100644 --- a/docs/modules/ROOT/pages/observation/introduction.adoc +++ b/docs/modules/ROOT/pages/observation/introduction.adoc @@ -45,4 +45,46 @@ include::{include-java}/observation/ObservationTestingTests.java[tags=example,in TIP: Calling `observe(() -> ...)` leads to starting the Observation, putting it in scope, running the lambda, putting an error on the Observation if one took place, closing the scope and stopping the Observation. -// TODO: Add a link to docs gen +[[micrometer-observation-building-blocks]] +== Building Blocks + +In this section we will describe a high overview of Micrometer Observation components. You can read more about those in xref:observation/components.adoc[this part of the documentation]. + +Glossary + +- `ObservationRegistry` - registry containing `Observation` related configuration (e.g. handlers, predicates, filters) +- `ObservationHandler` - handles lifecycle of an `Observation` (e.g. create a timer on observation start, stop it on observation stop) +- `ObservationFilter` - mutates the `Context` before the `Observation` gets stopped (e.g. add a high cardinality key-value with the cloud server region) +- `ObservationPredicate` - condition to disable creation of an `Observation` (e.g. don't create observations with given key-value) +- `Context` (actually `Observation.Context`) - a mutable map attached to an `Observation`, passed between handlers (that way you can pass state without doing any thread locals) +- `ObservationConvention` - mean to separate Observation lifecycle (starting, stopping, opening scopes) from adding meta-data (such as observation name, key-value pairs). That way the naming of observations and meta-data handling becomes a configuration problem (e.g. adding key-values do not require changing instrumentation code, but changing the convention) + +[[micrometer-observation-usage]] +=== Usage Flow + +// https://arthursonzogni.com/Diagon/#GraphDAG +// ObservationHandler->ObservationRegistry +// ObservationFilter->ObservationRegistry +// ObservationPredicate->ObservationRegistry +// ObservationRegistry->Observation +// Context->Observation +// ObservationConvention->Observation +[source,subs=+attributes] +----- +┌──────────────────┐┌─────────────────┐┌────────────────────┐┌───────┐┌─────────────────────┐ +│ObservationHandler││ObservationFilter││ObservationPredicate││Context││ObservationConvention│ +└┬─────────────────┘└┬────────────────┘└┬───────────────────┘└┬──────┘└┬────────────────────┘ +┌▽───────────────────▽──────────────────▽┐ │ │ +│ObservationRegistry │ │ │ +└┬───────────────────────────────────────┘ │ │ +┌▽────────────────────────────────────────────────────────────▽────────▽┐ +│Observation │ +└───────────────────────────────────────────────────────────────────────┘ +----- + +* `ObservationHandler`, `ObservationFilter`, `ObservationPredicate` get registered on the `ObservationRegistry` +** `ObservationHandler` can be composed together (e.g. via the `FirstMatchingCompositeObservationHandler` - that can be useful when you group multiple handlers of the same type, e.g. `TracingHandler` or `MeterHandler`) +* `ObservationRegistry`, `Context` are mandatory to create an `Observation` +** Either you create an `Observation` with name, `ObservationRegistry` and `Context` +** Or with `ObservationRegistry`, `ObservationConvention` +* When `Observation` calls its lifecycle methods, all `ObservationHandlers` that pass the `supportsContext` check with the corresponding `Context` will have their lifecycle methods executed (e.g. if a handler has an `instanceof SenderContext` check in `supportsContext` then it will be called only for such types of contexts) diff --git a/docs/src/test/java/io/micrometer/docs/observation/ObservationConfiguringTests.java b/docs/src/test/java/io/micrometer/docs/observation/ObservationConfiguringTests.java index 2c0e6ef266..90c06d0444 100644 --- a/docs/src/test/java/io/micrometer/docs/observation/ObservationConfiguringTests.java +++ b/docs/src/test/java/io/micrometer/docs/observation/ObservationConfiguringTests.java @@ -26,7 +26,7 @@ import static org.assertj.core.api.BDDAssertions.then; /** - * Sources for observation-handler.adoc + * Sources for observation-components.adoc */ class ObservationConfiguringTests { diff --git a/docs/src/test/java/io/micrometer/docs/observation/ObservationHandlerTests.java b/docs/src/test/java/io/micrometer/docs/observation/ObservationHandlerTests.java index b03be0d4e6..fea4717707 100644 --- a/docs/src/test/java/io/micrometer/docs/observation/ObservationHandlerTests.java +++ b/docs/src/test/java/io/micrometer/docs/observation/ObservationHandlerTests.java @@ -36,7 +36,7 @@ import static io.micrometer.docs.observation.ObservationHandlerTests.TaxObservationDocumentation.TaxLowCardinalityKeyNames.TAX_TYPE; /** - * Sources for observation-handler.adoc + * Sources for observation-components.adoc */ class ObservationHandlerTests {