GH-50622: [Docs][Format] Align Variant `typed_value` primitive type mappings with the Parquet shredding spec by sdf-jkl · Pull Request #50810 · apache/arrow

sdf-jkl · 2026-08-04T20:46:16Z

Rationale for this change

The existing Arrow spec is misleading and allows certain Arrow types to be typed value Variant fields. The Parquet Shredding spec specifically rejects them - Unsigned integers, Null.

Decimal mapping needs clarification about precision.

What changes are included in this PR?

Change the Primitive Type Mappings table for Parquet Variant extension type.

Rendered version - https://github.com/sdf-jkl/arrow/blob/gh50622-variant-typed-value-mappings/docs/source/format/CanonicalExtensions.rst#primitive-type-mappings

Are these changes tested?

N/A

Are there any user-facing changes?

Changes to the Arrow docs

Closes #50622

GitHub Issue: [Docs][Format] Variant extension: align typed_value primitive type mappings with the Parquet shredding spec #50622

github-actions · 2026-08-04T20:46:52Z

⚠️ GitHub issue #50622 has been automatically assigned in GitHub to PR creator.

sdf-jkl · 2026-08-13T15:33:54Z

Please take a look @alamb @wgtmac @zeroshade

alamb

Thank you @sdf-jkl -- this looks very nice to me except for a few small details I left comments on

I don't think we should merge this without making sure we ping the mailing list about it (perhaps again) as I think it could be interpreted as changing the spec.

alamb · 2026-08-17T19:16:27Z

+only valid Parquet representation.
+
+----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| Variant Type                           | Parquet Type                                     | Arrow ``typed_value`` Type                  |


Perhaps we could also add a reference to the location of the parquet and arrow type lists

I think the arrow ones are here:
https://arrow.apache.org/docs/format/Columnar.html#data-types

The parquet ones are here:
https://parquet.apache.org/docs/file-format/types/
https://parquet.apache.org/docs/file-format/types/logicaltypes/

I am a little confused about what (logical or physical) type is used in the Parquet Type column. Some seem to be just the physical type (INT64) , and some seem to have both logical and physical (INT32, INT(8, true) ). It would help to be spcific about what the Parquet type column represents and use it consistently

alamb · 2026-08-17T19:20:16Z

+----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| double                                 | DOUBLE                                           | Float64                                     |
+----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| decimal4 (1 <= P <= 9, 0 <= S <= P)    | INT32, DECIMAL(P, S)                             | Decimal32(P, S)                             |


https://parquet.apache.org/docs/file-format/types/variantencoding/ says

The Decimal type contains a scale, but no precision. The implied precision of a decimal value is floor(log_10(val)) + 1.

I don't unersatnd how that maps to 1 <= P <= 9, 0 <= S <= P 🤔

There's a small separate table for decimals that shows what precision can fit in the underlying physical type of a typed_value.

https://github.com/apache/parquet-format/blob/3ef43e00cea6dcd06da5b77541f84bef879aff5b/VariantEncoding.md?plain=1#L440-L447

alamb · 2026-08-17T19:22:20Z

+| uuid                                   | FIXED_LEN_BYTE_ARRAY[len=16], UUID               | :ref:`UUID extension type <uuid_extension>` |
+----------------------------------------+--------------------------------------------------+---------------------------------------------+
+
+The decimal precision bands follow the `Variant encoding types


I am still a little confused about how decimals are working -- is the this text talking about each variant value itself (as the scale/precision can vary row to row) or the min/max precision of all values in a column (presumably what is needed when shredding) 😕

It's talking about the constraint of physical types of the typed_value field, so it would be the latter. I'll polish it to make it more clear.

sdf-jkl · 2026-08-18T16:23:27Z

@alamb I think it's more clear now.

zeroshade · 2026-08-18T19:59:48Z

+   * A Variant null is always encoded in the ``value`` field (as ``00``),
+     never in ``typed_value``: a null ``typed_value`` signals that the row is
+     not shredded, and for shredded object fields a null ``typed_value``
+     together with a null ``value`` means the field is missing.


Should we clarify that a variant null should still be indicated in the null validity bitmap of the corresponding array (accompanying the 00 value in the value field)?

I don't think it should.

The spec explicitly says:

The value field must be annotated as required for unshredded Variant values, or optional if parts of the value are shredded as typed Parquet columns.

link

That makes Variant::Null and SQL NULL different things

I think it might be more precise to say it makes Variant::Null and a null value in the corresponding arrow array different things

alamb · 2026-08-19T11:40:35Z

+----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| int16                                  | INT32                             | INT(16, true)            | Int16                                       |
+----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| int32                                  | INT32                             |                          | Int32                                       |


i am confused that there is no logical type for int32, though this is the same in the parquet variant table too https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types

alamb · 2026-08-19T11:42:46Z

+   * A Variant null is always encoded in the ``value`` field (as ``00``),
+     never in ``typed_value``: a null ``typed_value`` signals that the row is
+     not shredded, and for shredded object fields a null ``typed_value``
+     together with a null ``value`` means the field is missing.


I think it might be more precise to say it makes Variant::Null and a null value in the corresponding arrow array different things

alamb

Thank you @sdf-jkl -- I think this makes sense to me. I'll wait for more confirmation from other community members before merging this in

pin the typed value typed to corresponding Parquet types

3afdca0

sdf-jkl requested a review from pitrou as a code owner August 4, 2026 20:46

github-actions Bot added Component: Documentation awaiting review Awaiting review labels Aug 4, 2026

sdf-jkl mentioned this pull request Aug 4, 2026

[Docs][Format] Variant extension: align typed_value primitive type mappings with the Parquet shredding spec #50622

Open

alamb reviewed Aug 17, 2026

View reviewed changes

sdf-jkl added 3 commits August 18, 2026 10:09

split parquet physical and logical type columns

683dc27

clean up decimal precision notes

cd26764

clean up pass

97fc879

zeroshade reviewed Aug 18, 2026

View reviewed changes

github-actions Bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Aug 18, 2026

alamb reviewed Aug 19, 2026

View reviewed changes

alamb approved these changes Aug 19, 2026

View reviewed changes

Uh oh!

Conversation

sdf-jkl commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Aug 4, 2026

Uh oh!

sdf-jkl commented Aug 13, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sdf-jkl Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sdf-jkl commented Aug 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sdf-jkl commented Aug 4, 2026 •

edited

Loading

sdf-jkl Aug 18, 2026 •

edited

Loading