Refactor Severless Gateway Inferred Span #9388
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
PerfectSlayer
added
tag: do not merge
|
🎯 Code Coverage 🔗 Commit SHA: e8c2baf | Docs | Was this helpful? Give us feedback! |
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 46 metrics, 13 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.53.0-SNAPSHOT~e8c2bafaff, baseline=1.53.0-SNAPSHOT~5aa25baa32
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.064 s) : 0, 1064413
Total [baseline] (10.821 s) : 0, 10821240
Agent [candidate] (1.046 s) : 0, 1045506
Total [candidate] (10.713 s) : 0, 10712996
section appsec
Agent [baseline] (1.222 s) : 0, 1222480
Total [baseline] (10.776 s) : 0, 10775761
Agent [candidate] (1.222 s) : 0, 1221646
Total [candidate] (10.781 s) : 0, 10781298
section iast
Agent [baseline] (1.179 s) : 0, 1178621
Total [baseline] (10.871 s) : 0, 10871406
Agent [candidate] (1.186 s) : 0, 1186454
Total [candidate] (10.939 s) : 0, 10938822
section profiling
Agent [baseline] (1.193 s) : 0, 1193475
Total [baseline] (10.813 s) : 0, 10813034
Agent [candidate] (1.196 s) : 0, 1196020
Total [candidate] (10.851 s) : 0, 10851198
gantt
title petclinic - break down per module: candidate=1.53.0-SNAPSHOT~e8c2bafaff, baseline=1.53.0-SNAPSHOT~5aa25baa32
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.483 ms) : 0, 1483
crashtracking [candidate] (1.444 ms) : 0, 1444
BytebuddyAgent [baseline] (744.385 ms) : 0, 744385
BytebuddyAgent [candidate] (731.412 ms) : 0, 731412
GlobalTracer [baseline] (246.513 ms) : 0, 246513
GlobalTracer [candidate] (242.433 ms) : 0, 242433
AppSec [baseline] (30.726 ms) : 0, 30726
AppSec [candidate] (30.194 ms) : 0, 30194
Debugger [baseline] (6.144 ms) : 0, 6144
Debugger [candidate] (6.082 ms) : 0, 6082
Remote Config [baseline] (682.53 µs) : 0, 683
Remote Config [candidate] (668.655 µs) : 0, 669
Telemetry [baseline] (13.084 ms) : 0, 13084
Telemetry [candidate] (12.152 ms) : 0, 12152
section appsec
crashtracking [baseline] (1.443 ms) : 0, 1443
crashtracking [candidate] (1.464 ms) : 0, 1464
BytebuddyAgent [baseline] (754.934 ms) : 0, 754934
BytebuddyAgent [candidate] (754.423 ms) : 0, 754423
GlobalTracer [baseline] (235.29 ms) : 0, 235290
GlobalTracer [candidate] (234.915 ms) : 0, 234915
AppSec [baseline] (167.534 ms) : 0, 167534
AppSec [candidate] (168.318 ms) : 0, 168318
Debugger [baseline] (9.534 ms) : 0, 9534
Debugger [candidate] (8.027 ms) : 0, 8027
Remote Config [baseline] (631.414 µs) : 0, 631
Remote Config [candidate] (637.073 µs) : 0, 637
Telemetry [baseline] (8.44 ms) : 0, 8440
Telemetry [candidate] (9.272 ms) : 0, 9272
IAST [baseline] (23.561 ms) : 0, 23561
IAST [candidate] (23.494 ms) : 0, 23494
section iast
crashtracking [baseline] (1.449 ms) : 0, 1449
crashtracking [candidate] (1.465 ms) : 0, 1465
BytebuddyAgent [baseline] (850.523 ms) : 0, 850523
BytebuddyAgent [candidate] (857.224 ms) : 0, 857224
GlobalTracer [baseline] (232.901 ms) : 0, 232901
GlobalTracer [candidate] (234.938 ms) : 0, 234938
AppSec [baseline] (28.615 ms) : 0, 28615
AppSec [candidate] (26.686 ms) : 0, 26686
Debugger [baseline] (5.753 ms) : 0, 5753
Debugger [candidate] (6.593 ms) : 0, 6593
Remote Config [baseline] (601.341 µs) : 0, 601
Remote Config [candidate] (611.451 µs) : 0, 611
Telemetry [baseline] (8.377 ms) : 0, 8377
Telemetry [candidate] (8.223 ms) : 0, 8223
IAST [baseline] (29.367 ms) : 0, 29367
IAST [candidate] (29.582 ms) : 0, 29582
section profiling
crashtracking [baseline] (1.42 ms) : 0, 1420
crashtracking [candidate] (1.42 ms) : 0, 1420
BytebuddyAgent [baseline] (759.807 ms) : 0, 759807
BytebuddyAgent [candidate] (761.733 ms) : 0, 761733
GlobalTracer [baseline] (222.227 ms) : 0, 222227
GlobalTracer [candidate] (222.272 ms) : 0, 222272
AppSec [baseline] (29.874 ms) : 0, 29874
AppSec [candidate] (30.034 ms) : 0, 30034
Debugger [baseline] (6.29 ms) : 0, 6290
Debugger [candidate] (6.307 ms) : 0, 6307
Remote Config [baseline] (721.358 µs) : 0, 721
Remote Config [candidate] (714.603 µs) : 0, 715
Telemetry [baseline] (16.401 ms) : 0, 16401
Telemetry [candidate] (16.263 ms) : 0, 16263
ProfilingAgent [baseline] (107.234 ms) : 0, 107234
ProfilingAgent [candidate] (107.594 ms) : 0, 107594
Profiling [baseline] (107.856 ms) : 0, 107856
Profiling [candidate] (108.24 ms) : 0, 108240
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.53.0-SNAPSHOT~e8c2bafaff, baseline=1.53.0-SNAPSHOT~5aa25baa32
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.043 s) : 0, 1043136
Total [baseline] (8.605 s) : 0, 8604806
Agent [candidate] (1.051 s) : 0, 1050848
Total [candidate] (8.632 s) : 0, 8631864
section iast
Agent [baseline] (1.197 s) : 0, 1196515
Total [baseline] (9.33 s) : 0, 9329693
Agent [candidate] (1.181 s) : 0, 1181006
Total [candidate] (9.35 s) : 0, 9350307
gantt
title insecure-bank - break down per module: candidate=1.53.0-SNAPSHOT~e8c2bafaff, baseline=1.53.0-SNAPSHOT~5aa25baa32
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.452 ms) : 0, 1452
crashtracking [candidate] (1.452 ms) : 0, 1452
BytebuddyAgent [baseline] (731.276 ms) : 0, 731276
BytebuddyAgent [candidate] (733.961 ms) : 0, 733961
GlobalTracer [baseline] (241.885 ms) : 0, 241885
GlobalTracer [candidate] (243.463 ms) : 0, 243463
AppSec [baseline] (29.965 ms) : 0, 29965
AppSec [candidate] (30.464 ms) : 0, 30464
Debugger [baseline] (6.071 ms) : 0, 6071
Debugger [candidate] (6.109 ms) : 0, 6109
Remote Config [baseline] (679.288 µs) : 0, 679
Remote Config [candidate] (680.337 µs) : 0, 680
Telemetry [baseline] (10.813 ms) : 0, 10813
Telemetry [candidate] (13.683 ms) : 0, 13683
section iast
crashtracking [baseline] (1.481 ms) : 0, 1481
crashtracking [candidate] (1.452 ms) : 0, 1452
BytebuddyAgent [baseline] (865.025 ms) : 0, 865025
BytebuddyAgent [candidate] (851.72 ms) : 0, 851720
GlobalTracer [baseline] (234.969 ms) : 0, 234969
GlobalTracer [candidate] (233.523 ms) : 0, 233523
AppSec [baseline] (27.281 ms) : 0, 27281
AppSec [candidate] (25.45 ms) : 0, 25450
Debugger [baseline] (6.588 ms) : 0, 6588
Debugger [candidate] (5.83 ms) : 0, 5830
Remote Config [baseline] (606.878 µs) : 0, 607
Remote Config [candidate] (599.855 µs) : 0, 600
Telemetry [baseline] (9.225 ms) : 0, 9225
Telemetry [candidate] (8.392 ms) : 0, 8392
IAST [baseline] (30.109 ms) : 0, 30109
IAST [candidate] (32.986 ms) : 0, 32986
LoadParameters
See matching parameters
SummaryFound 1 performance improvements and 2 performance regressions! Performance is the same for 9 metrics, 12 unstable metrics.
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.53.0-SNAPSHOT~e8c2bafaff, baseline=1.53.0-SNAPSHOT~5aa25baa32
dateFormat X
axisFormat %s
section baseline
no_agent (4.346 ms) : 4295, 4396
. : milestone, 4346,
iast (9.645 ms) : 9486, 9804
. : milestone, 9645,
iast_FULL (13.939 ms) : 13663, 14216
. : milestone, 13939,
iast_GLOBAL (10.416 ms) : 10236, 10597
. : milestone, 10416,
profiling (8.771 ms) : 8636, 8907
. : milestone, 8771,
tracing (7.379 ms) : 7277, 7481
. : milestone, 7379,
section candidate
no_agent (4.37 ms) : 4318, 4421
. : milestone, 4370,
iast (9.167 ms) : 9018, 9315
. : milestone, 9167,
iast_FULL (13.825 ms) : 13547, 14102
. : milestone, 13825,
iast_GLOBAL (10.396 ms) : 10213, 10578
. : milestone, 10396,
profiling (8.916 ms) : 8777, 9054
. : milestone, 8916,
tracing (7.749 ms) : 7632, 7866
. : milestone, 7749,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.53.0-SNAPSHOT~e8c2bafaff, baseline=1.53.0-SNAPSHOT~5aa25baa32
dateFormat X
axisFormat %s
section baseline
no_agent (36.728 ms) : 36427, 37030
. : milestone, 36728,
appsec (48.322 ms) : 47891, 48752
. : milestone, 48322,
code_origins (45.791 ms) : 45398, 46185
. : milestone, 45791,
iast (44.063 ms) : 43703, 44424
. : milestone, 44063,
profiling (48.315 ms) : 47863, 48767
. : milestone, 48315,
tracing (43.912 ms) : 43553, 44271
. : milestone, 43912,
section candidate
no_agent (37.201 ms) : 36904, 37497
. : milestone, 37201,
appsec (48.465 ms) : 48042, 48888
. : milestone, 48465,
code_origins (46.555 ms) : 46152, 46958
. : milestone, 46555,
iast (43.907 ms) : 43533, 44280
. : milestone, 43907,
profiling (50.101 ms) : 49607, 50595
. : milestone, 50101,
tracing (44.751 ms) : 44366, 45136
. : milestone, 44751,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics. Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.53.0-SNAPSHOT~e8c2bafaff, baseline=1.53.0-SNAPSHOT~5aa25baa32
dateFormat X
axisFormat %s
section baseline
no_agent (14.99 s) : 14990000, 14990000
. : milestone, 14990000,
appsec (15.005 s) : 15005000, 15005000
. : milestone, 15005000,
iast (18.444 s) : 18444000, 18444000
. : milestone, 18444000,
iast_GLOBAL (18.295 s) : 18295000, 18295000
. : milestone, 18295000,
profiling (15.298 s) : 15298000, 15298000
. : milestone, 15298000,
tracing (15.049 s) : 15049000, 15049000
. : milestone, 15049000,
section candidate
no_agent (15.54 s) : 15540000, 15540000
. : milestone, 15540000,
appsec (15.044 s) : 15044000, 15044000
. : milestone, 15044000,
iast (18.778 s) : 18778000, 18778000
. : milestone, 18778000,
iast_GLOBAL (17.998 s) : 17998000, 17998000
. : milestone, 17998000,
profiling (15.083 s) : 15083000, 15083000
. : milestone, 15083000,
tracing (14.789 s) : 14789000, 14789000
. : milestone, 14789000,
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.53.0-SNAPSHOT~e8c2bafaff, baseline=1.53.0-SNAPSHOT~5aa25baa32
dateFormat X
axisFormat %s
section baseline
no_agent (1.473 ms) : 1462, 1485
. : milestone, 1473,
appsec (2.417 ms) : 2367, 2467
. : milestone, 2417,
iast (2.199 ms) : 2136, 2262
. : milestone, 2199,
iast_GLOBAL (2.236 ms) : 2173, 2299
. : milestone, 2236,
profiling (2.047 ms) : 1997, 2097
. : milestone, 2047,
tracing (2.025 ms) : 1976, 2074
. : milestone, 2025,
section candidate
no_agent (1.473 ms) : 1462, 1485
. : milestone, 1473,
appsec (3.671 ms) : 3454, 3888
. : milestone, 3671,
iast (2.197 ms) : 2134, 2260
. : milestone, 2197,
iast_GLOBAL (2.24 ms) : 2177, 2303
. : milestone, 2240,
profiling (2.042 ms) : 1991, 2093
. : milestone, 2042,
tracing (2.023 ms) : 1974, 2072
. : milestone, 2023,
|
* Creates inferred proxy spans for API Gateway calls via presence of http headers --------- Co-authored-by: Zarir Hamza <[email protected]>
PerfectSlayer
force-pushed
the
bbujon/gateway-inferred-span2
branch
from
8e4e88b to
ad6356f
Compare
Avoid duplicate expensive context extraction Avoid subclassing tracing span for serverless but used serverless context element instead to store / track inferred span while keep tracing feature untouched Improved propagator to not create / capture inferred span context element on invalid data Rework context element to hold the inferred spans and its captured data Release captured data as soon as they span start (never read after this point so reclaiming memory) Refactor context element and propagator into the right package, not context component (product / feature agnostic) Refactor unit tests
PerfectSlayer
force-pushed
the
bbujon/gateway-inferred-span2
branch
from
ad6356f to
e8c2baf
Compare
PerfectSlayer
requested review from
AlexeyKuznetsov-DD
and removed request for
a team
Comment on lines
+65
to
+70
| long startTime; | ||
| try { | ||
| startTime = Long.parseLong(header(PROXY_START_TIME_MS)) * 1000; // Convert to microseconds | ||
| } catch (NumberFormatException e) { | ||
| return extracted; // Invalid timestamp | ||
| } |
There was a problem hiding this comment.
I have a feeling that we need to have some sort of utility code for such places?
WDYT?
Something like: long startTime = parseLong((header(PROXY_START_TIME_MS), ?some_default? ) * 1000;
There was a problem hiding this comment.
I don't think you can have a default start time in this case.
Gateway inferred spans represent the duration spent in upstream proxy servers (that are not instrumented for distributed tracing, they only inject few Datadog headers).
The start time represents the timestamp the request was intercepted by the proxy server and will be use to create a "proxy span" into the downstream service.
So if you don't have a valid start time, it's most likely a proxy issue, and you can't really "guess" when the proxy intercepted the upstream request.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
This PR continues @zarirhamza work about Serverless gateway inferred span in #8336 and brings the following improvements:
Motivation
Restore Serverless feature while meeting design and performance requirements
Additional Notes
This requires to complete the instrumentation refactoring first.
I already did a lot of them here:
The last one is pending for review:
beforeFinishMigration #9422Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: LANGPLAT-680