branch-4.0: [fix](regression) handle cumulative delete-version compaction wait #64945#65233
Open
shuke987 wants to merge 1 commit into
Open
branch-4.0: [fix](regression) handle cumulative delete-version compaction wait #64945#65233shuke987 wants to merge 1 commit into
shuke987 wants to merge 1 commit into
Conversation
…pache#64945) Fix `trigger_and_wait_compaction` so Cloud cumulative compaction that meets a delete version does not wait until the 300s timeout after valid progress has already happened. When cumulative compaction meets a delete version, BE can return `[E-2010] cumulative compaction meet delete version`, advance the cumulative point, and let base compaction handle the rowsets. In that path the cumulative success/failure timestamps may not change, so the old helper kept polling even after base compaction had completed and `run_status=false`. This patch treats `E-2010` plus cumulative point advancement plus a changed base success time as an equivalent completed cumulative delete-version path while still waiting when `run_status=true`. If `E-2010` advances the cumulative point but base success time has not changed yet, the helper keeps waiting even if the cumulative failure timestamp changed. The case `compaction/test_compacation_with_delete.groovy` creates alternating data and delete rowsets, then calls `trigger_and_wait_compaction(tableName, "cumulative")`. In Cloud mode this can legally follow: 1. cumulative compaction meets delete version and returns `E-2010` 2. cumulative point advances 3. base compaction handles the delete-version rowsets The helper only watched cumulative success/failure timestamp changes. In the failing log, base compaction completed in 448 ms, but the helper waited for 5 minutes because the cumulative timestamps did not change. - `git diff --check` - `git diff --check origin/master..HEAD` - Local Groovy condition simulation: - `E-2010 + cumulative point advanced + base success time changed + run_status=false` exits wait - `E-2010 + cumulative point advanced` keeps waiting if base success time has not changed - `E-2010 + cumulative point advanced + cumulative failure time changed` still keeps waiting if base success time has not changed - `run_status=true` keeps waiting even if the delete-version/base-success condition is met - normal cumulative success timestamp change still exits wait when there is no delete-version handoff Cloud P0 rerun is still needed for final validation. (cherry picked from commit 4f8c144)
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Collaborator
Author
|
run buildall |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Backport merged master PR #64945 to branch-4.0.
test_schema_change_with_empty_rowsetcallstrigger_and_wait_compaction(tableName, "cumulative")before the schema change. In Cloud mode, cumulative compaction can legally reportE-2010, advance the cumulative point, and let base compaction handle delete-version rowsets. The old helper can keep waiting for the cumulative timestamp path and hit the 300s timeout even after valid progress.This cherry-pick keeps the branch-4.0 HTTP retry behavior in the helper and adds the #64945 E-2010 completion condition.
Testing
git diff --check origin/branch-4.0..HEADrun buildall