Repository navigation
Compute writer chunk hints through the aggregate cache - #10266
connortsui20 wants to merge 1 commit into
Conversation
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Merging this PR will improve performance by 36.17%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | Simulation | bench_compare_sliced_dict_primitive[(2500, 10000)] |
83 µs | 42.8 µs | +94.17% |
| ⚡ | Simulation | chunked_opt_bool_canonical_into[(1000, 10)] |
69.9 µs | 59.3 µs | +17.81% |
| ⚡ | Simulation | compress_fsst[(500, 4, 8)] |
216.9 µs | 196.5 µs | +10.37% |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ct/stats-writer-cache-computation (aef451a) with ct/stats-remaining-consumers (019c756)
Footnotes
-
503 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Superseded by #10269, which groups this change with the related statistics work. The original commits and branch are preserved.
Original PR description
Tracking Issue: #10177
Summary
The file writer needs both a cached answer for each chunk and the partial state needed to combine chunks. A finalized answer cannot always supply that state: two individually sorted chunks can be out of order at their shared boundary. Publishing through a bound computation keeps both results tied to the input that produced them.
Changes
Reuse the writer's chunk accumulators through the aggregate cache API. Preserve fused extrema, cached Min/Max recovery, overflow state, and the existing omission of null chunk hints. For review, start with the bound computation contract, then its core accumulator dispatch, followed by the writer callers.
API Changes
Add
AggregationsRef::compute_into. It resets and computes a matching core accumulator, caches an exact non-null final, and leaves the real partial state available for merging.compute_resultcontinues to cache exact nulls.Stack
Depends on #10265 in migration stack #10229.