Improve logging of rebalancer and recovery #586
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mrForza
force-pushed
the
mrforza/gh-212-improvement-of-rebalancer-logging
branch
from
6b1057d to
64cc837
Compare
Serpentian
reviewed
Aug 20, 2025
Collaborator
Serpentian
left a comment
Serpentian
left a comment
There was a problem hiding this comment.
These are the comments for the first two commits, more comments are coming later) Thank you for working on this, good logging is crucial and allows us to investigate, what happened during incidents
Serpentian
reviewed
Aug 20, 2025
mrForza
force-pushed
the
mrforza/gh-212-improvement-of-rebalancer-logging
branch
3 times, most recently
from
5a8b3f8 to
f5c25f7
Compare
Serpentian
reviewed
Aug 25, 2025
Collaborator
Serpentian
left a comment
Serpentian
left a comment
There was a problem hiding this comment.
Oh, shit. I forgot to send the last message of review, I'm very sorry
Serpentian
reviewed
Aug 25, 2025
mrForza
force-pushed
the
mrforza/gh-212-improvement-of-rebalancer-logging
branch
2 times, most recently
from
04c506f to
ccff54f
Compare
Serpentian
requested changes
Sep 15, 2025
mrForza
force-pushed
the
mrforza/gh-212-improvement-of-rebalancer-logging
branch
3 times, most recently
from
46add65 to
a1c095b
Compare
Gerold103
reviewed
Nov 24, 2025
Collaborator
Gerold103
left a comment
Gerold103
left a comment
There was a problem hiding this comment.
Thanks for the patch! Great that you have taken this task. These logs really needed some polishing.
mrForza
force-pushed
the
mrforza/gh-212-improvement-of-rebalancer-logging
branch
2 times, most recently
from
9623c34 to
62c138e
Compare
mrForza
force-pushed
the
mrforza/gh-212-improvement-of-rebalancer-logging
branch
from
62c138e to
2ccab27
Compare
Serpentian
reviewed
Dec 8, 2025
mrForza
force-pushed
the
mrforza/gh-212-improvement-of-rebalancer-logging
branch
3 times, most recently
from
40bbd59 to
cc07222
Compare
Before this patch "Finish bucket recovery step ..." logs were printed at the end of recovery even if no buckets were successfully recovered. It led to unnecessary log records. This patch fixes the issue by adding an additional check for the number of recovered buckets. Part of tarantool#212 NO_DOC=bugfix
This patch introduces logging of buckets' ids which were recovered during recovery stage of storage. Part of tarantool#212 NO_DOC=bugfix
mrForza
force-pushed
the
mrforza/gh-212-improvement-of-rebalancer-logging
branch
from
cc07222 to
b91d290
Compare
This patch adds rebalancer routes' logging. The log file now includes information about the source storage, the number of buckets, and the destination storage where the buckets will be moved. Since the rebalancer service has changed logging of routes that were sent, we change the `rebalancer/rebalancer.test.lua` and `rebalancer/stress_add_remove_several_rs.test.lua` tests. Part of tarantool#212 NO_DOC=bugfix
Before this patch the function `rebalancer_request_state` returned only nil in case of errors (e.g. presence of SENDING, RECEIVING, GARBAGE buckets, not active rebalancer). This makes it inconsistent compared to other rebalancer functions. Now, in case of errors we return `(nil, err)` instead of `nil`. It can help us to propagate a meaningful error in rebalancer service. In addition we throw a new vshard error for cases when the storage has some non-active buckets during rebalancing and for case when rebalancer is not active or in progress. Also we remove "Some buckets are not active" log from `rebalancer_service_f` because a meaningful error about non-active buckets is already returned from `rebalancer_download_states`. NO_TEST=refactoring NO_DOC=refactoring
Before this patch the function `rebalancer_download_states` didn't return information about replicaset from which the states could not be downloaded. As a result, the errors returned from `rebalancer_download_states` lack of valuable information about unhealthy replicaset. Now, we add replicaset.id into error which is returned from the `rebalancer_download_states`. It can help us to propagate replicaset.id to rebalancer service. Closes tarantool#212 NO_DOC=bugfix
mrForza
force-pushed
the
mrforza/gh-212-improvement-of-rebalancer-logging
branch
from
b91d290 to
14d9c65
Compare
Gerold103
approved these changes
Dec 17, 2025
Collaborator
Gerold103
left a comment
Gerold103
left a comment
There was a problem hiding this comment.
🚀, lets also see if @Serpentian still has any comments unsolved.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before this patch "Finish bucket recovery step ..." logs were printed at
the end of recovery even if no buckets were successfully recovered, it led
to unnecessary log entries. This patch fixes the issue by adding an
additional check for the number of recovered buckets.
Closes #212
NO_DOC=bugfix