Add Karafka integration #4147

nvh0412 · 2024-11-22T05:09:57Z

What does this PR do?

In this PR, we introduce the Kafka integration for the Karafka gem. Which includes:

Distributed tracing by utilizing message.metadata.headers
Traces for worker.process and each message executor inside the batch, so we can link them to the origin trace if the distributed tracing is on

Motivation:

We’re integrating Karafka to implement proper distributed tracing in our system with Datadog, as it lacks an official integration. This integration will also enable distributed tracing if the message headers include distributed tracing data.

Distributed tracing will help us create a proper service map, connecting Kafka producers and consumers.

Change log entry

Yes. Add Karafka integration for distributed tracing.

(Added by @ivoanjo)

Additional Notes:

How to test the change?

docs/GettingStarted.md

lib/datadog/tracing/contrib/karafka/configuration/settings.rb

drichards-87 · 2024-11-22T20:14:59Z

Created a Jira card for Docs Team editorial review.

lib/datadog/tracing/contrib/karafka/events/worker/process.rb

appraisal/ruby-2.7.rb

appraisal/ruby-3.0.rb

lib/datadog/tracing/contrib/karafka/events/worker/process.rb

lib/datadog/tracing/contrib/karafka/integration.rb

codecov-commenter · 2024-11-22T23:31:00Z

Codecov Report

Attention: Patch coverage is 80.58252% with 20 lines in your changes missing coverage. Please review.

Project coverage is 97.75%. Comparing base (ce4393e) to head (27e5711).
Report is 41 commits behind head on master.

Files with missing lines	Patch %	Lines
lib/datadog/tracing/contrib/karafka/patcher.rb	51.61%	15 Missing ⚠️
lib/datadog/tracing/contrib/karafka.rb	80.00%	4 Missing ⚠️
lib/datadog/tracing/contrib/karafka/integration.rb	95.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4147      +/-   ##
==========================================
- Coverage   97.78%   97.75%   -0.03%     
==========================================
  Files        1353     1358       +5     
  Lines       81817    81920     +103     
  Branches     4145     4150       +5     
==========================================
+ Hits        80001    80081      +80     
- Misses       1816     1839      +23

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

mensfeld · 2024-11-24T18:42:41Z

FYI feel free to ping me once remarks are done. I will be happy to help and maybe in the future retire my own instrumentation in favour of the DD one ;)

lib/datadog/tracing/contrib/karafka/ext.rb

lib/datadog/tracing/contrib/karafka/patcher.rb

drichards-87

Left a couple of very small suggestions from Docs and approved the PR.

docs/GettingStarted.md

Use Tracing.trace wrapper to add consumer trace Use Instrumentation::Monitor to instrument and have a proper trace wrapper

…uns/13565626491

TonyCTHsu · 2025-02-27T12:33:54Z

Sorry, the pull request is currently blocked and I am working on

With Datadog's Github admin and security team to configure workflows to run properly in order to acquire the required status for merging pull requests.
Discussion with the maintainer about the minimum version to target for support.

This reverts commit 5446aaa.

TonyCTHsu · 2025-03-12T14:27:19Z

👋 @marco I have prepared a branch tmp/karafka-tracing to circumvent the security measures preventing secret access.

Basically, rebase tmp/karafka-tracing onto the branch of (karafka-tracing) of this pull request and push to trigger the CI. This will trigger our Gitlab pipeline, unit tests and system tests to acquire the required status check to be merged.

Currently, the unit tests are failing because of the minimum version defined subject to test. Static analysis is failing with rubocop and steep .

I did cherrypicked the changes containing autogenerated gemfiles and lockfiles. You might do the same in order to change your dependencies for testing.

taltcher · 2025-03-29T06:55:00Z

Hi - is there an ETA when this feature could be released? this is something we are really waiting for :-)

* master: (209 commits) Fix method override warning in profiler (DataDog#4549) Fix warnings in DI and tracing generated with -w (DataDog#4547) Add telemetry to tracking middleware patcher Improve devise instrumentation Add signature for devise tracking middleware Fix gateway push of identity.set_user Add prefix in ID extraction for Hash-like objects Push AppSec identity event into gateway Add debug logging into patches and middleware Switch into update system-tests Change login success/failure by set usr.login Add multi-user support in signin/signup tracking Add libddwaf into user tracking Adjust AppSec::Anonymizer interface Refactor Devise patches and middleware Add Devise::DataExtractor to extract id and login Update auto_user_instrumentation configuration Set AppSec collection mode tag in identity kit Add tests for SDK and automated instrumentation Add authenticated user tracking implementation ...

lib/datadog/tracing/contrib/karafka.rb

lib/datadog/tracing/contrib/karafka/integration.rb

p-datadog · 2025-04-01T08:05:47Z

lib/datadog/tracing/contrib/karafka/patcher.rb

+          def patch
+            require_relative 'monitor'
+
+            ::Karafka::App.config.monitor = Monitor.new


Could this be replacing another monitor added by the application?

https://github.com/karafka/karafka/wiki/Monitoring-and-logging appears to suggest a different pattern

@p-datadog not really: https://karafka.io/docs/Monitoring-and-Logging/#monitor-wrapping-and-replacement

both are supported and in this case there has to be a custom monitor to handle the tracing as far as I understand (karafka author here)

@mensfeld Thank you for the response. Can you clarify what happens when the monitor is replaced - is the original monitor going to continue receiving events?

I made a small change that patches the Karafka Monitor to invoke dd-trace-rb instrumentation which should work with any installed monitor. What is your opininion of this change?

@p-datadog when monitor is replaced it is replaced. You can use a monitor that wraps a different one and document it. 98-99% of people stick with the default one so it is in my opinion an edge case.

…uns/14199528606

p-datadog · 2025-04-01T15:39:05Z

karafka 2.0.41 (min version we are testing) uses Concurrent::Array without declaring a dependency on or requiring concurrent-ruby, which karafka-core 2.3.0 (that is being used currently by CI) does also not use in any way. I think the solution here is to pin to and older karafka-core version for karafka 2.0.41.

…uns/14200028580

mensfeld · 2025-04-01T17:58:40Z

karafka 2.0.41 (min version we are testing) uses Concurrent::Array without declaring a dependency on or requiring concurrent-ruby, which karafka-core 2.3.0 (that is being used currently by CI) does also not use in any way. I think the solution here is to pin to and older karafka-core version for karafka 2.0.41.

2.0.41 is totally EOL and making any decisions based on it does not make any sense: https://karafka.io/docs/Versions-Lifecycle-and-EOL/#karafka-components-support

this dependency reference was fixed later on. Concurrent ruby was completely removed in 2.3.

p-datadog · 2025-04-01T18:12:08Z

What minimum karafka version would you suggest we support? 2.3?

mensfeld · 2025-04-01T18:21:40Z

Yes. 2.3 is already EOL but it has all the dependency issues solved and a stable API.

github-actions bot added integrations Involves tracing integrations tracing labels Nov 22, 2024

Add Karafka integration

dd02868

nvh0412 force-pushed the karafka-tracing branch from 595ea74 to dd02868 Compare November 22, 2024 05:12

chore: Add karafka mapping

10645a7

nvh0412 marked this pull request as ready for review November 22, 2024 06:07

nvh0412 requested review from a team as code owners November 22, 2024 06:07

marcotc reviewed Nov 22, 2024

View reviewed changes

docs/GettingStarted.md Show resolved Hide resolved

marcotc reviewed Nov 22, 2024

View reviewed changes

lib/datadog/tracing/contrib/karafka/configuration/settings.rb Show resolved Hide resolved

drichards-87 added the editorial review Waiting for a review from the docs team label Nov 22, 2024

marcotc mentioned this pull request Nov 22, 2024

Add exception to instrumentation payload karafka/karafka-core#145

Closed

marcotc reviewed Nov 22, 2024

View reviewed changes

lib/datadog/tracing/contrib/karafka/events/worker/process.rb Outdated Show resolved Hide resolved

mensfeld reviewed Nov 22, 2024

View reviewed changes

appraisal/ruby-2.7.rb Outdated Show resolved Hide resolved

mensfeld reviewed Nov 22, 2024

View reviewed changes

appraisal/ruby-3.0.rb Outdated Show resolved Hide resolved

mensfeld reviewed Nov 22, 2024

View reviewed changes

lib/datadog/tracing/contrib/karafka/events/worker/process.rb Outdated Show resolved Hide resolved

mensfeld reviewed Nov 22, 2024

View reviewed changes

lib/datadog/tracing/contrib/karafka/integration.rb Show resolved Hide resolved

nvh0412 added 2 commits November 23, 2024 10:06

chore: remove redudnant changes

ecb1897

chore: Add karafka getting started properly

f9b6e36

mensfeld reviewed Nov 24, 2024

View reviewed changes

lib/datadog/tracing/contrib/karafka/ext.rb Show resolved Hide resolved

mensfeld reviewed Nov 24, 2024

View reviewed changes

lib/datadog/tracing/contrib/karafka/patcher.rb Show resolved Hide resolved

drichards-87 approved these changes Nov 25, 2024

View reviewed changes

docs/GettingStarted.md Outdated Show resolved Hide resolved

docs/GettingStarted.md Outdated Show resolved Hide resolved

nvh0412 added 3 commits November 26, 2024 23:31

chore: Address feedbacks

41cb65f

Use Tracing.trace wrapper to add consumer trace Use Instrumentation::Monitor to instrument and have a proper trace wrapper

chore: correct monitor require position

9dd469c

Merge branch 'master' into karafka-tracing

fbbbcb0

nvh0412 and others added 8 commits February 26, 2025 15:38

Fix failed specs

761e5b0

chore: correct rspec specs

531403e

chore: ignore Ruby 2.5 and 2.6 due to break changes in metadata.headers

c35d62d

Merge branch 'master' into karafka-tracing

4f557fa

Fix coverage matrix for karafka

b4166cc

Skip Ruby 2.5 for karfka-min

b6a95d9

[🤖] Lock Dependency: https://github.com/DataDog/dd-trace-rb/actions/r…

ba51ce6

…uns/13565626491

Fix min version

e34c2cb

TonyCTHsu added 4 commits March 10, 2025 13:13

Change to pull request target

5446aaa

Merge branch 'tonycthsu/karafka-community-support' into karafka-tracing

c3607a6

Revert "Change to pull request target"

4ab2fcf

This reverts commit 5446aaa.

Merge branch 'master' into karafka-tracing

8f945ac

p-datadog reviewed Apr 1, 2025

View reviewed changes

p and others added 7 commits April 1, 2025 07:20

fix reference

ff38761

simplify

c79769d

patch karafka monitor instead of replacing it

ffcef94

line length

2df3a6f

types

d0e5a43

Merge branch 'master' into karafka-tracing

9b64eff

[🤖] Lock Dependency: https://github.com/DataDog/dd-trace-rb/actions/r…

7b11928

…uns/14199528606

p and others added 2 commits April 1, 2025 11:42

try locking karafka-core version for karafka-min

5484090

[🤖] Lock Dependency: https://github.com/DataDog/dd-trace-rb/actions/r…

e788158

…uns/14200028580

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Karafka integration #4147

Add Karafka integration #4147

nvh0412 commented Nov 22, 2024 •

edited by Strech

Loading

drichards-87 commented Nov 22, 2024

codecov-commenter commented Nov 22, 2024 •

edited

Loading

mensfeld commented Nov 24, 2024

drichards-87 left a comment

TonyCTHsu commented Feb 27, 2025

TonyCTHsu commented Mar 12, 2025 •

edited

Loading

taltcher commented Mar 29, 2025

p-datadog Apr 1, 2025

p-datadog Apr 1, 2025

mensfeld Apr 1, 2025

p-datadog Apr 1, 2025

mensfeld Apr 1, 2025

p-datadog commented Apr 1, 2025

mensfeld commented Apr 1, 2025

p-datadog commented Apr 1, 2025

mensfeld commented Apr 1, 2025

Add Karafka integration #4147

Are you sure you want to change the base?

Add Karafka integration #4147

Conversation

nvh0412 commented Nov 22, 2024 • edited by Strech Loading

drichards-87 commented Nov 22, 2024

codecov-commenter commented Nov 22, 2024 • edited Loading

Codecov Report

mensfeld commented Nov 24, 2024

drichards-87 left a comment

Choose a reason for hiding this comment

TonyCTHsu commented Feb 27, 2025

TonyCTHsu commented Mar 12, 2025 • edited Loading

taltcher commented Mar 29, 2025

p-datadog Apr 1, 2025

Choose a reason for hiding this comment

p-datadog Apr 1, 2025

Choose a reason for hiding this comment

mensfeld Apr 1, 2025

Choose a reason for hiding this comment

p-datadog Apr 1, 2025

Choose a reason for hiding this comment

mensfeld Apr 1, 2025

Choose a reason for hiding this comment

p-datadog commented Apr 1, 2025

mensfeld commented Apr 1, 2025

p-datadog commented Apr 1, 2025

mensfeld commented Apr 1, 2025

nvh0412 commented Nov 22, 2024 •

edited by Strech

Loading

codecov-commenter commented Nov 22, 2024 •

edited

Loading

TonyCTHsu commented Mar 12, 2025 •

edited

Loading