Skip to content

Conversation

@watson
Copy link
Collaborator

@watson watson commented Feb 18, 2025

What does this PR do?

This is a workaround for, what seems like a bug in Node.js core, that seems to trigger when, among other things, a lot of timers are being created very rapidly. This makes the call to setTimeout throw an error from within AsyncLocalStorage._propagate with the following TypeError:

Cannot read properties of undefined (reading 'Symbol(kResourceStore)')

Source of the bug: https://github.com/nodejs/node/blob/v18.20.6/lib/async_hooks.js#L312

Motivation

Don't crash customer applications.

Plugin Checklist

Additional Notes

Copy link
Collaborator Author

watson commented Feb 18, 2025

This stack of pull requests is managed by Graphite. Learn more about stacking.

@github-actions
Copy link

github-actions bot commented Feb 18, 2025

Overall package size

Self size: 8.75 MB
Deduped: 94.95 MB
No deduping: 95.47 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | @datadog/libdatadog | 0.4.0 | 29.44 MB | 29.44 MB | | @datadog/native-appsec | 8.4.0 | 19.25 MB | 19.26 MB | | @datadog/native-iast-taint-tracking | 3.3.0 | 13.77 MB | 13.78 MB | | @datadog/pprof | 5.5.1 | 9.79 MB | 10.17 MB | | protobufjs | 7.2.5 | 2.77 MB | 5.16 MB | | @datadog/native-iast-rewriter | 2.8.0 | 2.6 MB | 2.74 MB | | @opentelemetry/core | 1.14.0 | 872.87 kB | 1.47 MB | | @datadog/native-metrics | 3.1.0 | 1.06 MB | 1.46 MB | | @opentelemetry/api | 1.8.0 | 1.21 MB | 1.21 MB | | import-in-the-middle | 1.11.2 | 112.74 kB | 835.4 kB | | source-map | 0.7.4 | 226 kB | 226 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | lru-cache | 7.18.3 | 133.92 kB | 133.92 kB | | pprof-format | 2.1.0 | 111.69 kB | 111.69 kB | | @datadog/sketches-js | 2.1.0 | 109.9 kB | 109.9 kB | | lodash.sortby | 4.7.0 | 75.76 kB | 75.76 kB | | ignore | 5.3.2 | 53.63 kB | 53.63 kB | | shell-quote | 1.8.1 | 44.96 kB | 44.96 kB | | istanbul-lib-coverage | 3.2.0 | 29.34 kB | 29.34 kB | | rfdc | 1.3.1 | 25.21 kB | 25.21 kB | | @isaacs/ttlcache | 1.4.1 | 25.2 kB | 25.2 kB | | tlhunter-sorted-set | 0.1.0 | 24.94 kB | 24.94 kB | | limiter | 1.1.5 | 23.17 kB | 23.17 kB | | dc-polyfill | 0.1.4 | 23.1 kB | 23.1 kB | | retry | 0.13.1 | 18.85 kB | 18.85 kB | | semifies | 1.0.0 | 15.84 kB | 15.84 kB | | jest-docblock | 29.7.0 | 8.99 kB | 12.76 kB | | crypto-randomuuid | 1.0.0 | 11.18 kB | 11.18 kB | | ttl-set | 1.0.0 | 4.61 kB | 9.69 kB | | path-to-regexp | 0.1.12 | 6.6 kB | 6.6 kB | | koalas | 1.0.2 | 6.47 kB | 6.47 kB | | module-details-from-path | 1.0.3 | 4.47 kB | 4.47 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@codecov
Copy link

codecov bot commented Feb 18, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.90%. Comparing base (172bc66) to head (22ab9c4).
Report is 272 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5290      +/-   ##
==========================================
+ Coverage   80.80%   80.90%   +0.09%     
==========================================
  Files         487      487              
  Lines       21710    21797      +87     
==========================================
+ Hits        17542    17634      +92     
+ Misses       4168     4163       -5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@datadog-datadog-prod-us1
Copy link

datadog-datadog-prod-us1 bot commented Feb 18, 2025

Datadog Report

Branch report: watson/DEBUG-3526/nodejs-core-bug-workaround
Commit report: bd7cd0a
Test service: dd-trace-js-integration-tests

✅ 0 Failed, 674 Passed, 0 Skipped, 16m 8.67s Total Time

@pr-commenter
Copy link

pr-commenter bot commented Feb 18, 2025

Benchmarks

Benchmark execution time: 2025-02-19 09:23:56

Comparing candidate commit 22ab9c4 in PR branch watson/DEBUG-3526/nodejs-core-bug-workaround with baseline commit 172bc66 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 915 metrics, 18 unstable metrics.

Copy link
Contributor

@cbeauchesne cbeauchesne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about adding failing system-tests scenarios?

it's here : https://github.com/DataDog/dd-trace-js/blob/master/.github/workflows/system-tests.yml#L34

Just add a line : scenarios: DEBUGGER_PII_REDACTION, DEBUGGER_PROBES_SNAPSHOT

@watson watson force-pushed the watson/DEBUG-3526/nodejs-core-bug-workaround branch from 70ceee5 to 4d83e30 Compare February 18, 2025 15:13
@watson watson requested a review from a team as a code owner February 18, 2025 15:13
This is a workaround for, what seems like a bug in Node.js core, that
seems to trigger when, among other things, a lot of timers are being
created very rapidly. This makes the call to `setTimeout` throw an error
from within `AsyncLocalStorage._propagate` with the following TypeError:

    Cannot read properties of undefined (reading 'Symbol(kResourceStore)')
@watson watson force-pushed the watson/DEBUG-3526/nodejs-core-bug-workaround branch from 4d83e30 to 75e555e Compare February 19, 2025 09:00
@watson watson requested a review from cbeauchesne February 19, 2025 09:00
cbeauchesne
cbeauchesne previously approved these changes Feb 19, 2025
@watson watson enabled auto-merge (squash) February 19, 2025 10:14
@watson watson merged commit 77abd1e into master Feb 19, 2025
386 of 388 checks passed
@watson watson deleted the watson/DEBUG-3526/nodejs-core-bug-workaround branch February 19, 2025 10:17
watson added a commit that referenced this pull request Feb 19, 2025
…5290)

This is a workaround for, what seems like a bug in Node.js core, that
seems to trigger when, among other things, a lot of timers are being
created very rapidly. This makes the call to `setTimeout` throw an error
from within `AsyncLocalStorage._propagate` with the following TypeError:

    Cannot read properties of undefined (reading 'Symbol(kResourceStore)')
@watson watson mentioned this pull request Feb 19, 2025
watson added a commit that referenced this pull request Feb 20, 2025
…5290)

This is a workaround for, what seems like a bug in Node.js core, that
seems to trigger when, among other things, a lot of timers are being
created very rapidly. This makes the call to `setTimeout` throw an error
from within `AsyncLocalStorage._propagate` with the following TypeError:

    Cannot read properties of undefined (reading 'Symbol(kResourceStore)')
@watson watson mentioned this pull request Feb 20, 2025
6 tasks
@watson watson added the bug Something isn't working label May 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants