Skip to content

[feature](inverted index) support DROP INDEX ON PARTITION for inverted index#62181

Merged
airborne12 merged 4 commits intoapache:masterfrom
airborne12:feature/drop-index-on-partition
Apr 9, 2026
Merged

[feature](inverted index) support DROP INDEX ON PARTITION for inverted index#62181
airborne12 merged 4 commits intoapache:masterfrom
airborne12:feature/drop-index-on-partition

Conversation

@airborne12
Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: close #xxx

Problem Summary:

Currently, DROP INDEX only supports table-level operations — it removes both the index metadata and physical index files across all partitions. In contrast, BUILD INDEX already supports partition-level operations via BUILD INDEX ON PARTITION.

This PR adds the corresponding inverse operation: DROP INDEX ON PARTITION, which deletes only the physical inverted index files on specified partitions while preserving the table-level index definition. This is useful for:

  • Cleaning up index data on specific partitions to reclaim storage space
  • Selectively removing indexes from cold/archive partitions while keeping them on hot partitions

Syntax:

-- Standalone form
DROP INDEX [IF EXISTS] idx_name ON table_name PARTITIONS (p1, p2, ...);

-- ALTER TABLE form
ALTER TABLE table_name DROP INDEX [IF EXISTS] idx_name PARTITION (p1);

Key design decisions:

  • Only inverted indexes are supported (other index types are rejected)
  • Table-level index metadata is NOT modified — the index definition remains in the schema
  • Queries on affected partitions automatically fallback to non-index scan via enable_fallback_on_missing_inverted_index (default: true)
  • Works in both storage-compute separation (cloud) and storage-compute integrated (local) modes
  • No BE-side or cloud meta service changes required — reuses existing IndexChangeJob and buildOrDeleteTableInvertedIndices infrastructure

Release note

Support DROP INDEX ON PARTITION for inverted indexes, allowing users to delete physical index files on specified partitions without removing the table-level index definition.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • Yes. Added new SQL syntax DROP INDEX ... PARTITION/PARTITIONS (...) that was previously not supported. Existing DROP INDEX behavior without partition spec is unchanged.
  • Does this need documentation?

    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

…d index

Add partition-level DROP INDEX support for inverted indexes. This allows
users to delete physical inverted index files on specified partitions
without removing the table-level index metadata definition.

Syntax:
  DROP INDEX idx_name ON table_name PARTITIONS (p1, p2);
  ALTER TABLE table_name DROP INDEX idx_name PARTITION (p1);

When partitions are specified, only the physical index files on those
partitions are deleted via IndexChangeJob. The index definition remains
in the table schema, and queries on affected partitions automatically
fallback to non-index scan via enable_fallback_on_missing_inverted_index.

This is the inverse of BUILD INDEX ON PARTITION and works in both
storage-compute separation (cloud) and storage-compute integrated (local)
modes without any BE-side or cloud meta service changes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Thearas
Copy link
Copy Markdown
Contributor

Thearas commented Apr 7, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@airborne12
Copy link
Copy Markdown
Member Author

run buildall

…d strengthen tests

1. P1: Reject PARTITIONS (*) in DropIndexOp.validate() — previously caused
   NPE when PartitionNamesInfo.getPartitionNames() returned null.
2. P1: Reject TEMPORARY PARTITION in DropIndexOp.validate() — previously
   silently accepted but no delete job was created (correctness bug).
3. P2: Strengthen regression tests — assert job count increment after DROP
   INDEX ON PARTITION, and verify with enable_fallback_on_missing_inverted_index=false
   that index files were actually deleted on dropped partitions.
4. P2: Add error cases for PARTITIONS (*) and TEMPORARY PARTITION.
5. Add FE unit tests covering validate() and SchemaChangeHandler.process()
   error paths.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@airborne12
Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen
Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 46.58% (34/73) 🎉
Increment coverage report
Complete coverage report

…tions

- Replace all order_qt_* queries with sql + assertEquals to avoid
  dependency on .out files
- Remove fallback=false exception test — INVERTED_INDEX_FILE_NOT_FOUND
  is a BE internal status code that doesn't propagate to SQL client
- Keep job count increment assertions as the reliable verification
  that DROP INDEX ON PARTITION jobs are actually created

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@airborne12
Copy link
Copy Markdown
Member Author

run buildall

…tion

NGRAM_BF CREATE INDEX goes through schema change in cloud mode,
not lightweight index change. Use wait_for_last_col_change_finish
instead of build_index_on_table to avoid race condition where
BUILD INDEX runs before schema change completes.
@airborne12
Copy link
Copy Markdown
Member Author

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 9, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 9, 2026

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 9, 2026

PR approved by anyone and no changes requested.

Copy link
Copy Markdown
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@airborne12 airborne12 merged commit 791c56d into apache:master Apr 9, 2026
31 of 33 checks passed
@airborne12 airborne12 deleted the feature/drop-index-on-partition branch April 9, 2026 13:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/4.1.x dev/4.1.x-conflict reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants