Improve logging of rebalancer and recovery #586
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mrForza
force-pushed
the
mrforza/gh-212-improvement-of-rebalancer-logging
branch
from
6b1057d to
64cc837
Compare
Serpentian
reviewed
Aug 20, 2025
Serpentian
reviewed
Aug 20, 2025
mrForza
force-pushed
the
mrforza/gh-212-improvement-of-rebalancer-logging
branch
3 times, most recently
from
5a8b3f8 to
f5c25f7
Compare
Serpentian
reviewed
Aug 25, 2025
Serpentian
reviewed
Aug 25, 2025
mrForza
force-pushed
the
mrforza/gh-212-improvement-of-rebalancer-logging
branch
2 times, most recently
from
04c506f to
ccff54f
Compare
Serpentian
requested changes
Sep 15, 2025
mrForza
force-pushed
the
mrforza/gh-212-improvement-of-rebalancer-logging
branch
3 times, most recently
from
46add65 to
a1c095b
Compare
mrForza
force-pushed
the
mrforza/gh-212-improvement-of-rebalancer-logging
branch
from
a1c095b to
1da8c2c
Compare
Serpentian
reviewed
Sep 19, 2025
mrForza
force-pushed
the
mrforza/gh-212-improvement-of-rebalancer-logging
branch
from
1da8c2c to
f07abe5
Compare
Serpentian
requested changes
Sep 30, 2025
Before this patch "Finish bucket recovery step ..." logs were printed at the end of recovery even if no buckets were successfully recovered. It led to unnecessary log records. This patch fixes the issue by adding an additional check for the number of recovered buckets. Part of tarantool#212 NO_DOC=bugfix
This patch introduces logging of buckets' ids which were recovered during recovery stage of storage. Part of tarantool#212 NO_DOC=bugfix
mrForza
force-pushed
the
mrforza/gh-212-improvement-of-rebalancer-logging
branch
2 times, most recently
from
871197d to
489b425
Compare
Serpentian
reviewed
Oct 8, 2025
| end) | ||
| t.assert(g.replica_1_a:grep_log( | ||
| 'Apply rebalancer routes with 1 workers')) | ||
| end) |
There was a problem hiding this comment.
Nit: the indent is not correct here
There was a problem hiding this comment.
fixed. I also changed indents on 211-214 lines.
vshard/storage/init.lua
Outdated
| end | ||
| log.info('Rebalance routes are sent. Schedule next wakeup after '.. | ||
| '%f seconds', consts.REBALANCER_WORK_INTERVAL) | ||
| log.info('Next rebalancer routes were sent: %s. Schedule next ' .. |
There was a problem hiding this comment.
Nit: sounds incorrect gramatically) Let's better say The following rebalancer routes were sent, or you can just leave as it was in order not to change the existing tests)
| g.replica_2_a:replicaset_uuid()) | ||
| t.assert(g.replica_1_a:grep_log(rebalancer_routes_msg)) | ||
| start_bucket_move(g.replica_1_a, g.replica_2_a, moved_bucket_from_2) | ||
| start_bucket_move(g.replica_1_a, g.replica_3_a, moved_bucket_from_3) |
There was a problem hiding this comment.
You're moving the buckets with rebalancer, why do you need to manually move them then?
This patch adds rebalancer routes' logging. The log file now includes information about the source storage, the number of buckets, and the destination storage where the buckets will be moved. Since the rebalancer service has changed logging of routes that were sent, we change the `rebalancer/rebalancer.test.lua` and `rebalancer/stress_add_remove_several_rs.test.lua` tests. Part of tarantool#212 NO_DOC=bugfix
Before this patch the function `rebalancer_download_states` didn't return information about replicaset from which the states could not be downloaded. As a result, the log "Some buckets are not active ..." lacks of valuable information about unhealthy replicaset. Now, we return `(replicaset.id, nil)` instead of `nil` in case when rebalancer can't download state from this replicaset. Also we add replicaset.id in "Some buckets are not active ..." log. Also we change `rebalancer/rebalancer.test.lua` test which expected the old "Some buckets are not active" log without replicaset.id. Closes tarantool#212 NO_DOC=bugfix
mrForza
force-pushed
the
mrforza/gh-212-improvement-of-rebalancer-logging
branch
from
489b425 to
fce8f28
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before this patch "Finish bucket recovery step ..." logs were printed at
the end of recovery even if no buckets were successfully recovered, it led
to unnecessary log entries. This patch fixes the issue by adding an
additional check for the number of recovered buckets.
Closes #212
NO_DOC=bugfix