Topic/christian/disconnect slow peers #54
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ckreibich
force-pushed
the
topic/christian/disconnect-slow-peers
branch
from
30af91e to
9d11c51
Compare
ckreibich
force-pushed
the
topic/christian/disconnect-slow-peers
branch
2 times, most recently
from
b4f29ac to
b622c6b
Compare
This adds re-peering at the Broker level for peers that Broker decided to unpeer. We keep this at the Broker level since this behavior is specific to it (as opposed to other cluster backends). Includes baseline updates for btests that pick up on the new script's @load.
This translates Broker IDs to cluster-level node names, if available.
This module is loaded by the telemetry framework, which we're now loading via the cluster framework, i.e. also in bare mode. The resulting additional thread (for creating reporter.log) trips up a number of btest baselines. version.zeek doesn't use any of the string helper functions.
This adds a Broker-specific script to the cluster framework, loaded only when
Zeek is running in cluster mode. It adds logging in cluster.log as well as
telemetry via a metrics counter for Broker-observed backpressure overflows. The
new zeek_broker_backpressure_overflows counter, labeled by the neighboring peer
that the reporting node has determined to be unresponsive, counts the number of
unpeerings.
Here the node "worker" has observed node "proxy" falling behind once:
# HELP zeek_broker_backpressure_overflows_total Number of Broker peering drops due to a neighbor falling too far behind in message I/O
# TYPE zeek_broker_backpressure_overflows_total counter
zeek_broker_backpressure_overflows_total{endpoint="worker",peer="proxy"} 1
Includes btest baseline updates.
Yes, really. :-) We've hit the need for this on occasion in very specific settings and always worked around it via ugly nested loops or similars. This has ample warning that folks normally won't want to use this.
ckreibich
force-pushed
the
topic/christian/disconnect-slow-peers
branch
from
b622c6b to
65129b9
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.