Skip to content

Return aggregate results from scan source statistics - #10253

Closed
connortsui20 wants to merge 1 commit into
ct/stats-file-aggregatesfrom
ct/stats-source-aggregates
Closed

connortsui20 wants to merge 1 commit into
ct/stats-file-aggregatesfrom
ct/stats-source-aggregates

Conversation

@connortsui20

@connortsui20 connortsui20 commented Oct 3, 2026 •

Copy link
Copy Markdown
Member

Superseded by #10265, which groups this change with the related statistics work. The original commits and branch are preserved.

Original PR description

Tracking Issue: #10177

Summary

The scan source contract still exposes StatsSet, which prevents the DataFusion V2 planner from consuming aggregate-based summaries. Returning finalized aggregate results makes this remaining source boundary consistent with file summaries.

Changes

Update the source trait, its layout implementations, and the DataFusion V2 consumer together. Conversion preserves exact and inexact values, keeps a known null Sum, and leaves missing or unrepresentable planner statistics unknown. Layout sources continue to return empty metadata without reading arrays.

For review, start with the source trait and the DataFusion converter, then follow the projected-field hookup.

API Changes

DataSource::field_statistics returns AggregateResults instead of StatsSet.

Stack

Migration stack #10229, in review order: #10226 → #10227 → #10232 → #10235 → #10236 → #10252 → #10253 → #10257 → #10258 → #10259 → #10260 → #10261. Independent fixes: file Sum overflow #10228 and empty Constant aggregate states #10254.

@connortsui20 connortsui20 added the changelog/break A breaking API change label Oct 3, 2026
@connortsui20
connortsui20 added this pull request to stack #10229 October 3, 2026 00:45
@codspeed

codspeed Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by 12.23%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 2102 untouched benchmarks
⏩ 503 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ Simulation compress_fsst[(200, 16, 8)] 182.5 µs 162.6 µs +12.23%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/stats-source-aggregates (4e9cde3) with ct/stats-file-aggregates (1169c46)

Open in CodSpeed

Footnotes

  1. 503 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/break A breaking API change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant