Blocked BitPacked: encode blocked bitwidths

[mhk197]

Tracking Issue: #10167

Summary

Add per-block bit-packing, mirroring the single-width API:

	Choose widths	Encode at given widths
single width	bitpack_to_best_bit_width(array, ctx)	bitpack_encode(array, bit_width, bit_width_freq, ctx)
blocked	bitpack_blocked_to_best_bit_widths(array, ctx)	bitpack_encode_blocked(array, bit_widths, num_exceptions, ctx)

BitPackedData::encode_blocked(array, bit_widths, ctx) sits alongside encode. Blocked output always stores its block offsets as a primitive array (BitWidths::Blocked). Choosing between uniform and blocked output is left to the caller.

The single-width encoder is unchanged apart from sharing the non-negative check with the blocked encoder.

Design

Each block's width minimizes its packed size plus the cost of its exceptions, using the existing best_bit_width cost model on a per-block histogram. The zero padding of a partial last block and null values count as zero-width values, so every block is charged for its full 128 * width bytes. Each block's cost depends only on its own width, and blocked output always pays for its offsets, so choosing every block independently is optimal under this model.

Block offsets use the narrowest unsigned type that holds the final offset.

Additionally, I made the choice to allow blocks that are not packed at all -- but only if there exists at least one block that IS packed. Otherwise, we could encounter a scenario where one block is unpacked and the rest are, and we reject on the basis of one block out of many. There are definitely some improvements that can be made in the future.

Scope

Nothing calls the new encoder yet. fastlanes.bitpacked serialization requires a constant width, so wiring it into BtrBlocks waits for the variable-width wire format.

Results

Measured locally on 64K u32 values with synthetic distributions, comparing bitpack_to_best_bit_widths with bitpack_to_best_bit_width. Sizes are uncompressed nbytes, from 262,144 raw bytes:

Distribution	Single width	Blocked	Change
uniform (7-bit)	57,344 (w=7)	57,474	+130 B (+0.2%)
drift (blocks cycle 1–16 bits)	131,072 (w=16)	69,892	−46.7%
outliers (7-bit, 66 values at 21 bits)	58,384 (w=7, 66 patches)	58,514	+130 B (+0.2%)
zero_heavy (1 in 4 blocks 12-bit)	98,304 (w=12)	24,706	−74.9%

On uniform and outlier data both encoders pack and patch identically. The 130 bytes are the materialized offsets (65 u16 values).

Median encode time, including choosing the widths (median of three local runs):

Distribution	Single width	Blocked
uniform	74.2 µs	78.2 µs
drift	78.7 µs	83.5 µs
outliers	124.2 µs	127.8 µs
zero_heavy	98.2 µs	97.3 µs

[codspeed]

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 2101 untouched benchmarks
🆕 3 new benchmarks
⏩ 518 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
🆕	WallTime	bitpack_blocked_compress_avx2	N/A	7.7 µs	N/A
🆕	WallTime	bitpack_blocked_compress_avx512	N/A	5.8 µs	N/A
🆕	WallTime	bitpack_blocked_compress_neon	N/A	11.4 µs	N/A

---

_{Comparing mk/bitpacked-variable-encode (3471916) with develop (d97fb1c)}

Footnotes

1. 518 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[joseph-isaacs]

bitpack_to_best_bit_width and bitpack_to_best_bit_widths should be more distinct

[mhk197]

@joseph-isaacs can do bitpack_blocked_to_best_bit_widths