Blocked BitPacked: store block offsets as a child

[mhk197]

Tracking Issue: #10167

Summary

Add an optional block_offsets child to BitPackedArray that stores the byte boundaries of independently sized packed blocks, and make the shared bit width in BitPackedData optional.

An array has exactly one of the two: a global bit width, as today, or block offsets.

Design

Bit widths

BitPackedData::global_bit_width is Some(width) when every block shares a width, and None when block_offsets is Some. Validation requires exactly one of them.

BitPackedArrayExt::bit_widths() replaces bit_width():

pub enum BitWidths {
    /// Every block is packed at this bit width.
    Global(u8),
    /// Byte boundaries of the packed blocks, from which each block's bit width is derived.
    Blocked(ArrayRef),
}

Paths that need one width (decoding, scalar_at, v1 serialization and CUDA) return an error for blocked arrays until the decoder follow-ups.

There is a viewed and owned version of this.

Block offsets

block_offsets is a non-nullable unsigned integer array with num_blocks + 1 entries, including a trailing end boundary.

Boundaries are relative to the first entry. Block i occupies bytes block_offsets[i] - block_offsets[0] to block_offsets[i + 1] - block_offsets[0] of the packed buffer, at (block_offsets[i + 1] - block_offsets[i]) / 128 bits, because a 1024-value block at width w takes 128 * w bytes.

For example, three blocks at widths 3, 0 and 7 have offsets [0, 384, 384, 1280]. Offsets with equal steps are still blocked; nothing converts them to a global width.

Validation

• A global width must be at most ptype.bit_width() (previously at most 64), and the packed buffer must hold num_blocks * 128 * width bytes, as before.
• Block offsets must have a non-nullable unsigned integer dtype and num_blocks + 1 entries. Host-resident primitive offsets are also checked boundary by boundary: they must be nondecreasing, each block must be a whole number of 128-byte rows at most ptype.bit_width() bits wide, and together they must span exactly the packed buffer. Other encodings and device-resident offsets only get the dtype and length checks.

Scope

Nothing in this PR produces block offsets, but arrays built with try_new_with_block_offsets can be constructed and inspected. Kernels decline them, and decoding, scalar access, serialization and CUDA return an error.

Wire format and compatibility

fastlanes.bitpacked is unchanged. A global-width array has no block offsets child, so it serializes exactly as before, and the reader builds a global-width array from the metadata bit_width. Serializing a blocked array with this plugin returns an error. Compression goldens are unchanged.

API changes

• BitPackedArrayExt::bit_width() and BitPackedData::bit_width() are replaced by BitPackedArrayExt::bit_widths() -> BitWidths.
• BitPackedDataParts::bit_width: u8 is replaced by bit_widths: BitWidths.
• BitPacked::try_new_with_block_offsets constructs an array from packed data and block boundaries. BitPacked::try_new and BitPackedData::try_new are unchanged.
• BitUnpackedChunks::try_new and both unpacked_chunks accessors return an error for blocked arrays.
• bitpack_decompress::unpack_single returns VortexResult<Scalar>.
• The unused max_packed_value() accessor is removed.
• The Python FastLanesBitPackedArray.bit_width property returns int | None.

[codspeed]

Merging this PR will degrade performance by 41.92%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 1 regressed benchmark
✅ 2142 untouched benchmarks
⏩ 476 skipped benchmarks¹
🗄️ 1 archived benchmark run²

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	take[small_m/shuffled/primitive/nonnull/chunks=2048/indices=64]	279.4 µs	481 µs	-41.92%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing mk/bitpacked-stack-03-width-child (9c01ab9) with develop (9a17a1d)}

Footnotes

1. 476 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. 1 benchmark was run, but is now archived. If it was deleted in another branch, consider rebasing to remove it from the report. Instead if it was added back, click here to restore it. ↩

[mhk197]

this is very expensive. It is necessary to do a full pass of blocked offsets to make sure the bitpacked array is valid unfortunately, since it depends on array values now as well as metadata

[mhk197]

Additionally, I expect this check to rarely run since the block offsets will be compressed

[joseph-isaacs]

make it a view?

Suggested change

	Blocked(ArrayRef),
	Blocked(&'a ArrayRef),

[joseph-isaacs]

or have a viewed version

[mhk197]

I'll make this into BitWidthsView enum and add an owned BitWidths enum for when we need to do into_parts

[joseph-isaacs]

So we don't need this clone