Repository navigation
Return aggregate results from scan source statistics - #10253
connortsui20 wants to merge 1 commit into
Conversation
Merging this PR will improve performance by 12.23%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | Simulation | compress_fsst[(200, 16, 8)] |
182.5 µs | 162.6 µs | +12.23% |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ct/stats-source-aggregates (4e9cde3) with ct/stats-file-aggregates (1169c46)
Footnotes
-
503 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
00b0dad to
1169c46
Compare
c85b231 to
4e9cde3
Compare
Superseded by #10265, which groups this change with the related statistics work. The original commits and branch are preserved.
Original PR description
Tracking Issue: #10177
Summary
The scan source contract still exposes
StatsSet, which prevents the DataFusion V2 planner from consuming aggregate-based summaries. Returning finalized aggregate results makes this remaining source boundary consistent with file summaries.Changes
Update the source trait, its layout implementations, and the DataFusion V2 consumer together. Conversion preserves exact and inexact values, keeps a known null Sum, and leaves missing or unrepresentable planner statistics unknown. Layout sources continue to return empty metadata without reading arrays.
For review, start with the source trait and the DataFusion converter, then follow the projected-field hookup.
API Changes
DataSource::field_statisticsreturnsAggregateResultsinstead ofStatsSet.Stack
Migration stack #10229, in review order: #10226 → #10227 → #10232 → #10235 → #10236 → #10252 → #10253 → #10257 → #10258 → #10259 → #10260 → #10261. Independent fixes: file Sum overflow #10228 and empty Constant aggregate states #10254.