Improve document grouping in fabric_doc_update#6032
Open
nickva wants to merge 1 commit into
Open
Conversation
nickva
force-pushed
the
fix-groups-by-range-in-fabric-doc-update
branch
from
bb78d62 to
8b33ae2
Compare
nickva
force-pushed
the
fix-groups-by-range-in-fabric-doc-update
branch
from
8b33ae2 to
1a1a408
Compare
nickva
force-pushed
the
fix-groups-by-range-in-fabric-doc-update
branch
from
1a1a408 to
14039a4
Compare
nickva
force-pushed
the
fix-groups-by-range-in-fabric-doc-update
branch
from
14039a4 to
69342d9
Compare
The idea of doc/shard grouping and rotation was that different databases would start their workers on different copies, spread those uniformly and consistently by database and range. However, the order of entries were actually dependent on `dict:to_list/1` internals. So, if `dict` was changed to return sorted items, like for instance maps do, we'd end up suddenly breaking the supposedly randomized spread (I found this out by actually trying to use a map there instead of a dict). To fix this, make the ordering explicit. Group the entries first, then order each range's group by the unified `mem3:owners/3` membership order, so the first copy of each range lands on the owner node, consistent with "Unify membership hashes", and the result no longer depends on dict internals at all. While at it make it so we don't have to duplicate the main rotate function in tests and use the actual function we intend to test.
nickva
force-pushed
the
fix-groups-by-range-in-fabric-doc-update
branch
from
69342d9 to
c36d65f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The idea of doc/shard grouping and rotation was that different databases would start their workers on different copies, spread those uniformly and consistently by
{Range, DbName}.However, the order of entries were actually dependent on
dict:to_list/1internals. So, ifdictwas changed to return sorted items, like for instance maps do, we'd end up suddently breaking the supposedly randomized spread (I found this out by actually trying to use a map there instead of a dict).To fix this, make the rotation explicit. Group the entries first, then sort each group, and then rotate by
{Range, DbName}.While at it make it so we don't have to duplicate the main rotate function in tests and use the actual function we intend to test.