Migrate statistics consumers and file summaries to aggregates

[connortsui20]

Tracking Issue: #10177

Summary

Introducing a generic cache only helps if the existing callers use it. This migrates array operations, file summaries, pruning, and engine integrations together so they agree on aggregate identity, options, and precision.

Historical files already contain both array-node hints and footer summaries. Compatibility adapters translate those fields into finalized aggregate results and project representable results back onto the existing wire format. This keeps existing files readable without requiring a new edition or rewriting the data.

Changes

• Move aggregate helpers, array transformations, casts, and compression callers onto aggregate results.
• Replace fixed-stat file selection and lookup with aggregate functions and immutable summaries.
• Accumulate file summaries through aggregate states, retaining string truncation and directional bounds.
• Update pruning, DataFusion, DuckDB, JNI, scan sources, displays, and IPC consumers.

For review, start with stats/compat and layouts/file_stats, then the footer and pruning paths. The remaining changes migrate callers to those APIs. The residual legacy facade will be removed as the remaining callers and tests are converted.

API Changes

File summary selection and lookup now accept aggregate functions. stats_sets() and fixed-stat file lookups are replaced by aggregate result access. The historical FlatBuffers fields remain unchanged.

Stack

1. Introduce aggregate results and array caching #10235: aggregate cache and contracts.
2. This PR: callers and persisted summaries.
3. Preserve aggregate facts from writers and array producers #10269: populate aggregate metadata from writers and producers.

[codspeed]

Merging this PR will degrade performance by 12.76%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
❌ 2 regressed benchmarks
✅ 2100 untouched benchmarks
⏩ 503 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	filter_powerlaw_by_mostly_true[250000]	104 µs	150.3 µs	-30.82%
❌	Simulation	chunked_opt_bool_canonical_into[(1000, 10)]	59.7 µs	69.9 µs	-14.64%
⚡	Simulation	compress_fsst[(200, 16, 8)]	181.5 µs	161.4 µs	+12.46%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ct/stats-remaining-consumers (019c756) with ct/stats-aggregate-transfer (54d91e0)}

Footnotes

1. 503 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩