Skip to content

feat(parquet): add statistics-based row group filtering - #956

Open
YinZheng-Sun wants to merge 5 commits into
apache:mainfrom
YinZheng-Sun:parquet-row-group-filter-reader
Open

YinZheng-Sun wants to merge 5 commits into
apache:mainfrom
YinZheng-Sun:parquet-row-group-filter-reader

Conversation

@YinZheng-Sun

@YinZheng-Sun YinZheng-Sun commented Sep 22, 2026

Copy link
Copy Markdown

What

Adds ParquetMetricsRowGroupFilter to evaluate Iceberg predicates against Parquet footer statistics before reading row groups.

Adds tests covering split intersection, non-contiguous row groups, physical row positions, unprojected filter columns, schema evolution, missing statistics, predicate binding, and delete handling.

Why

The Parquet reader previously selected row groups only by file split. Filter predicates could not eliminate row groups whose footer statistics ruled out matching rows.

Behavior change

  • Enables statistics-based pruning by default through read.parquet.row-group-filter.enabled.
  • Retained row groups still require residual predicate evaluation; this does not implement exact row filtering.
  • Batches stop at row-group boundaries, including when pruning is disabled.

Testing

Built and ran parquet_test, data_test, avro_test, and expression_test locally during implementation. Rebuilt and reran parquet_test and data_test after the binding and field-mapping refactors; both passed.

The IN regression covers 201 input literals that deduplicate to 200 values, ensuring the limit applies to the bound set.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant