Skip to content

Refine BitPackingScheme to per-block bit widths when fastlanes.bitpacked.v2 is allowed - #10245

Open
mhk197 wants to merge 24 commits into
mk/bitpacked-v2-wirefrom
mk/bitpacked-v2-scheme
Open

mhk197 wants to merge 24 commits into
mk/bitpacked-v2-wirefrom
mk/bitpacked-v2-scheme

Conversation

@mhk197

@mhk197 mhk197 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Tracking Issue: #10167

Stacked on #10244; this PR targets mk/bitpacked-v2-wire.

Summary

BitPackingScheme packs each 1024-element block at its own bit width when the compressor's allowed serialized IDs include fastlanes.bitpacked.v2, following FoRScheme (#10136). Default writes are unchanged: the core edition allows only fastlanes.bitpacked, so refine keeps v1.

BitPackingScheme

  • BitPackingScheme gains v1 and v2 modes with v1() and v2() constructors, and Default is v1. The session registers BITPACKING_V1.
  • refine returns v2 when fastlanes.bitpacked.v2 is allowed, and v1 otherwise.
  • produced_encodings: v1 declares fastlanes.bitpacked; v2 declares fastlanes.bitpacked.v2.
  • Compress: v1 is unchanged. v2 always produces a blocked array, like FoRScheme v2 always encodes per-chunk references. It chooses every block's width with bitpack_to_best_bit_widths (Blocked BitPacked: encode blocked bitwidths #10203), even when every block chooses the same width, and compresses the block offsets as child 0 with compress_child. Patches are handled as in v1: compressed in place, or moved into Patched under experimental patches.
  • Estimate: unchanged. Both modes sample, and the sampler stitches 64-value runs from across the array, so sampled blocks don't resemble real ones and the sample tends toward one width. That is conservative for v2.

FoRScheme still bit-packs its encoded child with BITPACKING_V1, so FoR arrays keep a global width for now.

CUDA

The CUDA preset excludes fastlanes.bitpacked.v2, as it does fastlanes.for.v2. CUDA decoding returns an error for per-block bit widths.

Testing

  • refine picks v2 exactly when the v2 ID is allowed, from either mode.
  • tests/bitpacking_config.rs: with only BitPacking and every ID allowed, both blocks needing 1 to 8 bits and uniform 7-bit blocks serialize as v2, and nullable values round trip. The core edition never writes v2, and the CUDA preset writes v1.

Follow-ups

mhk197 added 20 commits October 2, 2026 15:05
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
…ked.v2 is allowed

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
@mhk197 mhk197 added the changelog/feature A new feature label Oct 2, 2026
@mhk197
mhk197 added this pull request to stack #10243 October 2, 2026 22:11
@codspeed

codspeed Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 2112 untouched benchmarks
⏩ 503 skipped benchmarks1


Comparing mk/bitpacked-v2-scheme (c98cb99) with mk/bitpacked-v2-wire (336bbfc)

Open in CodSpeed

Footnotes

  1. 503 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

mhk197 added 4 commits October 2, 2026 18:15
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant