Epic: Refactor Statistics #10175

This Epic follows #7707. It tracks the remaining migration from legacy statistics to aggregate functions and carries forward the work already completed there.

Status

Active. Aggregate computation, aggregate-based pruning, and zoned summaries have landed. Array caches, file footers, and compressor analysis still have legacy APIs to migrate, and we potentially want to rethink how in-memory stats are propagated/cached/tracked.

Goal

We want to use aggregate APIs consistently for all of statistics computation, caching, stored summaries, and downstream consumers.

First, we need to deprecate/remove the runtime ArrayStats, StatsSet, and fixed Stat APIs while preserving file compatibility.

After that, we need to figure out what other things we need to do. Future work might include array-cache and footer migration, engine integration updates, compressor statistics, and migration documentation. Nested footer coverage continues under #6389 and #9775.

Motivation

The aggregate system already supports extensible statistics, but the remaining legacy APIs require separate mappings, storage, and propagation rules. Completing the migration lets new statistics use the same contracts across computation and consumers.

Finalized results, conservative bounds, and mergeable partial states need distinct treatment throughout this migration.

Unresolved questions

• Should reusable results remain array-owned, move to an explicit input owner, or use another lifetime? Do we even need them? What do they give us, and what are the tradeoffs?
• Can array-owned caching be removed without losing useful shortcuts or introducing equivalent costs elsewhere?
• How should producer guarantees such as sortedness reach kernels without requiring a scan?
• Which rich compressor analyses should be retained as typed aggregate states (because at least right now it is slightly different from the other stats we have in vortex-array, for example)?

Implementation history

• Aggregate Fns #6721 introduced aggregate functions, followed by aggregate implementations for the existing statistics.
• Centralize aggregate stat bridge #7931 bridged aggregates to the legacy array cache.
• Add stats rewrite session API #7930 introduced the statistics rewrite session API.
• Add pruning aggregate functions #8025 added aggregates for pruning.
• Add built-in stats rewrite rules #7935 introduced aggregate-based pruning rules for built-in functions.
• Use aggregate descriptors for zoned stats #7938 migrated zone summaries to aggregate descriptors and partial states.
• Rework AggregateVTable #9816 clarified typed partial-state contracts.