Stateful logs blackhole #1568
Draft
Stateful logs blackhole #1568
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
blt
force-pushed
the
blt/wip_datadog_intake
branch
14 times, most recently
from
95a304e to
1e47b6e
Compare
TheSafo
force-pushed
the
stateful-logs-blackhole
branch
from
d6bfdbb to
1643968
Compare
blt
force-pushed
the
blt/wip_datadog_intake
branch
5 times, most recently
from
a603e60 to
ecc65f7
Compare
TheSafo
force-pushed
the
stateful-logs-blackhole
branch
2 times, most recently
from
25e5a3e to
b61a8ce
Compare
blt
force-pushed
the
blt/wip_datadog_intake
branch
from
ecc65f7 to
33f62f6
Compare
This commit introduces an accumulator mechanism into the capture manager, allowing for a rolling-window delay of metrics. The goal here is to support historic metric uses cases, such as that present in a theoretical Datadog intake blackhole, necessary to support #1546 and #1530. Signed-off-by: Brian L. Troutwine <[email protected]>
This commit removes from discovered or user-provided labels any reserved fields. We do this to avoid a serialization bug where a label duplicates one of lading's always-present fields. Signed-off-by: Brian L. Troutwine <[email protected]>
Signed-off-by: Brian L. Troutwine <[email protected]>
This commit adds an additional fuzz harness to assert that capture files produced by lading_capture are valid. This runs the CaptureManager in afl++ in forking mode, meaning we exercise the global registry, historical metrics et al without needing to elide any part of the novel machinery here. This fuzz test is however very slow. We rely on tokio's pause/advance to manage time but it's still not quick to run. I have not found any validation issues in produced capture files. Signed-off-by: Brian L. Troutwine <[email protected]>
Signed-off-by: Brian L. Troutwine <[email protected]>
Signed-off-by: Brian L. Troutwine <[email protected]>
Signed-off-by: Brian L. Troutwine <[email protected]>
Signed-off-by: Brian L. Troutwine <[email protected]>
This commit corrects a draining data overwrite bug which caused the last minute of data to be constant. This is corrected by introducing an additional guard interval that prevents overwrites, we guard against the behavior with a new test in state_machine. Signed-off-by: Brian L. Troutwine <[email protected]>
blt
force-pushed
the
blt/wip_datadog_intake
branch
from
33f62f6 to
8e4638b
Compare
I have also left notes for myself in this commit to follow-up on in the next few. Signed-off-by: Brian L. Troutwine <[email protected]>
Signed-off-by: Brian L. Troutwine <[email protected]>
blt
force-pushed
the
blt/wip_datadog_intake
branch
from
8e4638b to
d79fa1f
Compare
Signed-off-by: Brian L. Troutwine <[email protected]>
Signed-off-by: Brian L. Troutwine <[email protected]>
Signed-off-by: Brian L. Troutwine <[email protected]>
blt
force-pushed
the
blt/wip_datadog_intake
branch
from
d79fa1f to
35e80aa
Compare
Signed-off-by: Brian L. Troutwine <[email protected]>
blt
force-pushed
the
blt/wip_datadog_intake
branch
from
35e80aa to
80a1af2
Compare
Signed-off-by: Brian L. Troutwine <[email protected]>
This PR introduces support for Datadog intake v2 in a new blackhole, the goal being to accumulate trace-agent metrics that are only transmitted to dogstatsd and then out to the intake. The construction method here is intended to make it easy to add support for new intake versions. Signed-off-by: Brian L. Troutwine <[email protected]>
blt
force-pushed
the
blt/wip_datadog_intake
branch
from
80a1af2 to
a5a433e
Compare
Signed-off-by: Brian L. Troutwine <[email protected]>
Signed-off-by: Brian L. Troutwine <[email protected]>
TheSafo
force-pushed
the
stateful-logs-blackhole
branch
from
b61a8ce to
7542753
Compare
blt
force-pushed
the
blt/wip_datadog_intake
branch
2 times, most recently
from
420a3a5 to
cae011d
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
A brief description of the change being made with this pull request.
Motivation
What inspired you to submit this pull request?
Related issues
A list of issues either fixed, containing architectural discussions, otherwise relevant
for this Pull Request.
Additional Notes
Anything else we should know when reviewing?