[bobhan1]

Background

Compaction records a mapping from each physical input row to its output segment and row ID. MoW delete bitmap conversion and inverted-index merging consume this mapping; its dense representation stores two uint32_t values per input row, including filtered rows.

What problem does this PR solve?

Issue Number: None
Related PR: None
PR Type: Performance

Problem Summary:

Large mappings can consume gigabytes and repeatedly fail compaction allocations. This PR compresses eligible mappings during construction with fixed 4096-source-row blocks, enabling compression by default when the estimated dense row-pair array strictly exceeds 64 MiB. Both MoW bitmap conversion and Duplicate Key/MoW SNII postings compaction can use it. Small mappings and consumers requiring the dense matrix keep DENSE.

Details

• Change map: 1,614 production lines changed, 827 test lines, 61 documentation lines, and no generated files. The codec/storage kernel accounts for 666 production lines; the facade and concrete storage alternatives for 594; SNII validation/posting adaptation for 213; selection, bitmap errors, configuration and build integration for the rest. Much of the facade change moves existing inline dense/lazy storage logic into the implementation file.
• Why it was slow: Memory is the limiting resource: dense row pairs cost 8 bytes per physical source row, or 8 GB per billion rows, before other compaction allocations. Repeated allocation failures can prevent compaction from finishing.
• Optimization: Seal fixed source-row blocks while merging, without depending on a maximum segment row count. EMPTY/LINEAR omit redundant values, DELTA uses varint64 with beneficial Zstd level 1, and RAW64 caps payload expansion. A shared 1 MiB paged arena and 32-byte block entries bound allocation overhead. Finalization releases builders/encoder and derives 64-bit destination prefixes from actual output segment counts. Source cursors retain one decoded block each and share a decoder workspace per executing SNII plan. Storage ownership uses composition with a private std::variant: one dispatch surrounds each append batch, with no per-row mode branches or virtual calls. Dense consumers retain direct spans and CLucene's existing matrix API.
• Results: Four synthetic 262,144-row kernel benchmarks use 1/16 sources, approximately 10% missing rows and two gap distributions. The dense array alone is 2,097,152 bytes; the compressed directory is 2,048 bytes in every case, and interleaved sequential scans decode exactly 64 blocks.

Sources	Gap parameter	Encoded payload (bytes)	Retained kernel allocation (bytes)	Tracked overall peak (bytes)
1	1	20,273	1,050,832	1,470,944
1	16	185,936	1,050,832	1,470,944
16	1	20,704	1,051,792	1,963,424
16	16	229,915	1,051,792	1,963,424

Retained allocation includes directory capacity, the full arena page and kernel metadata. The tracker includes transient allocations and reader workspaces, but these are requested bytes, not RSS. Exclusions include allocator bookkeeping, benchmark input/baseline copies, enclosing source indexes/destination prefixes and other compaction state. A billion-row EMPTY-source test separately verifies its 7,812,512-byte directory; it is not a billion-row compression run. These small benchmarks explicitly construct the compressed kernel and bypass automatic mode selection.

ASAN/O0 observations: build 67.2–74.5 ms, cold decode 5.1–10.5 ms, sequential lookup 83.8–93.0 ms, 4,096 grouped sparse lookups 31.3–409.8 ms and 4,096 random lookups 226.9–475.8 ms. Direct dense scans take 3.0–3.9 ms. Sequential UT timing includes assertions; these are not optimized production or end-to-end compaction results.

• Applicability and cost: Compaction::should_compress_rowid_conversion() checks consumer/order constraints and the threshold before any source registration. It counts physical rows in actual data inputs, including rows later deleted/filtered, excluding whole changelog rowsets already removed. Comparing against a remaining row budget avoids multiplying or summing huge counts. CLucene, cluster keys, sequence maps and legacy index debug dumps remain DENSE. Random seeks may repeatedly decode blocks; per-source workspaces and arena tail space can dominate small maps. The 64 MiB default is an initial size policy, not a production-measured break-even point. No spill or external CLucene changes are introduced.
• Behavior change: enable_compressed_rowid_conversion=true and compressed_rowid_conversion_threshold_bytes=67108864 are dynamic BE defaults. Exactly the threshold remains DENSE; zero compresses every eligible nonempty mapping and negative values are rejected. Changes affect subsequent selections, not mappings already being built. The compaction profile records the selected RowIdConversion mode. Bitmap visibility and SNII bounds/monotonicity/uniqueness/full coverage validation remain intact; conversion errors propagate as Status. Persistent formats and FE/BE protocols are unchanged.

Testing

Locally, 83/83 ASAN_UT tests passed (7 suites):

bash run-be-ut.sh --run \
  --filter='CompressedRowIdMapTest.*:CompressedRowIdMapBenchmark.*:SniiRowIdConversionTest.*:SniiPostingCursorTest.*:SniiIndexCompactionTest.*:*TestRowIdConversion*' \
  -j100

Coverage includes threshold boundaries, multiple data inputs, zero/disabled settings, huge row counts, rejected negative thresholds, dense-only exclusions and Duplicate Key SNII selection. Compressed merge-reader tests cover Duplicate Key/MoW, horizontal/vertical merging, overlapping/nonoverlapping/mixed segments and delete predicates. Additional cases verify all three storage lifecycles, codec corruption/allocation failure, physical segment IDs, bitmap visibility, SNII posting layouts and byte-identical in-memory index rebuilds. The existing parameterized DENSE reader suite also passes.

Repository clang-format 16 checks on changed files, build-support/check-build-hygiene.sh and git diff --check passed. Actual local/cloud compaction, concurrent loading, lock-hold time and optimized production timing remain separate validation work. clang-tidy was not run.

Notes for Reviewers

BlockMeta uses scoped Kind/Flag enums and explicit flag_bits() conversion for masks. Inline ASCII diagrams document the 32-byte directory and four payload layouts; matching encoder/decoder examples explain bitmap holes, delta-minus-one varints and optional Zstd wrapping. The in-memory layout is unchanged.

See be/test/storage/rowid_conversion_benchmark.md for configuration, accounting boundaries and reproduction. The single arena tail in a small benchmark must not be extrapolated as another page for every 262,144 rows in a larger shared map.

The cloud/config.h include in segment_read_ahead.cpp makes its direct is_cloud_mode dependency explicit after unity-batch changes. The codec source is excluded from unity compilation to keep Zstd's static-workspace feature macro local.

Release note

Eligible MoW and Duplicate Key SNII compactions now compress row-ID mappings when their estimated dense array exceeds 64 MiB. The dynamic compressed_rowid_conversion_threshold_bytes setting controls this threshold; setting enable_compressed_rowid_conversion=false retains dense mappings.

Check List (For Author)

• Behavior changed: Yes (see Details)
• Does this need documentation: Yes; configuration and developer reproduction are documented in the in-tree benchmark guide. User documentation is planned with actual compaction validation.
• Pre-submission status: Partially verified: targeted ASAN unit tests pass; actual compaction and optimized performance validation remain pending.

AI Disclosure

• Tool: OpenAI Codex (GPT-6)
• Generated: Implementation, unit tests, benchmark, developer guide and PR description.
• Human review: Pending.

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?