Skip to content

GH-50622: [Docs][Format] Align Variant typed_value primitive type mappings with the Parquet shredding spec #50810

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sdf-jkl wants to merge 1 commit into
base: main
Choose a base branch
from
+77 −58
Open

GH-50622: [Docs][Format] Align Variant typed_value primitive type mappings with the Parquet shredding spec #50810

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
135 changes: 77 additions & 58 deletions docs/source/format/CanonicalExtensions.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -450,7 +450,7 @@ binary values look like.
* A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
(unshredded variants consist of just the ``metadata`` and ``value`` fields only)

* A field named ``typed_value`` which can be a :ref:`variant_primitive_type_mapping` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
* A field named ``typed_value`` which can be any Arrow type listed in the :ref:`variant_primitive_type_mapping` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``

* If the ``typed_value`` field is a ``List``, ``LargeList`` or ``ListView`` its elements **must** be *non-nullable* and **must**
be a ``Struct`` consisting of at least one (or both) of the following:
Expand Down Expand Up @@ -488,63 +488,82 @@ binary values look like.
Primitive Type Mappings
-----------------------

+----------------------+------------------------+
| Arrow Primitive Type | Variant Primitive Type |
+======================+========================+
| Null | Null |
+----------------------+------------------------+
| Boolean | Boolean (true/false) |
+----------------------+------------------------+
| Int8 | Int8 |
+----------------------+------------------------+
| Uint8 | Int16 |
+----------------------+------------------------+
| Int16 | Int16 |
+----------------------+------------------------+
| Uint16 | Int32 |
+----------------------+------------------------+
| Int32 | Int32 |
+----------------------+------------------------+
| Uint32 | Int64 |
+----------------------+------------------------+
| Int64 | Int64 |
+----------------------+------------------------+
| Float | Float |
+----------------------+------------------------+
| Double | Double |
+----------------------+------------------------+
| Decimal32 | decimal4 |
+----------------------+------------------------+
| Decimal64 | decimal8 |
+----------------------+------------------------+
| Decimal128 | decimal16 |
+----------------------+------------------------+
| Date32 | Date |
+----------------------+------------------------+
| Time64 | TimeNTZ |
+----------------------+------------------------+
| Timestamp(us, UTC) | Timestamp (micro) |
+----------------------+------------------------+
| Timestamp(us) | TimestampNTZ (micro) |
+----------------------+------------------------+
| Timestamp(ns, UTC) | Timestamp (nano) |
+----------------------+------------------------+
| Timestamp(ns) | TimestampNTZ (nano) |
+----------------------+------------------------+
| Binary | Binary |
+----------------------+------------------------+
| LargeBinary | Binary |
+----------------------+------------------------+
| BinaryView | Binary |
+----------------------+------------------------+
| String | String |
+----------------------+------------------------+
| LargeString | String |
+----------------------+------------------------+
| StringView | String |
+----------------------+------------------------+
| UUID extension type | UUID |
+----------------------+------------------------+
The following table defines the set of Arrow types that are valid as primitive
``typed_value`` storage. It is derived from the `Shredded Value Types
<https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types>`__
table of the Parquet Variant Shredding specification: each row maps a Variant
primitive type to the Parquet type required for a shredded ``typed_value``
column (physical type, followed by the logical type annotation if any) and to
the Arrow type(s) able to represent that Variant type's full value domain.
A ``typed_value`` field of one of the listed Arrow types holds values of
exactly the corresponding Variant type, and the listed Parquet type is its
only valid Parquet representation.

+----------------------------------------+--------------------------------------------------+---------------------------------------------+
| Variant Type | Parquet Type | Arrow ``typed_value`` Type |

@alamb alamb Aug 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps we could also add a reference to the location of the parquet and arrow type lists

I think the arrow ones are here:
https://arrow.apache.org/docs/format/Columnar.html#data-types

The parquet ones are here:
https://parquet.apache.org/docs/file-format/types/
https://parquet.apache.org/docs/file-format/types/logicaltypes/

I am a little confused about what (logical or physical) type is used in the Parquet Type column. Some seem to be just the physical type (INT64) , and some seem to have both logical and physical (INT32, INT(8, true) ). It would help to be spcific about what the Parquet type column represents and use it consistently

+========================================+==================================================+=============================================+
| boolean | BOOLEAN | Boolean |
+----------------------------------------+--------------------------------------------------+---------------------------------------------+
| int8 | INT32, INT(8, true) | Int8 |
+----------------------------------------+--------------------------------------------------+---------------------------------------------+
| int16 | INT32, INT(16, true) | Int16 |
+----------------------------------------+--------------------------------------------------+---------------------------------------------+
| int32 | INT32 | Int32 |
+----------------------------------------+--------------------------------------------------+---------------------------------------------+
| int64 | INT64 | Int64 |
+----------------------------------------+--------------------------------------------------+---------------------------------------------+
| float | FLOAT | Float32 |
+----------------------------------------+--------------------------------------------------+---------------------------------------------+
| double | DOUBLE | Float64 |
+----------------------------------------+--------------------------------------------------+---------------------------------------------+
| decimal4 (1 <= P <= 9, 0 <= S <= P) | INT32, DECIMAL(P, S) | Decimal32(P, S) |

@alamb alamb Aug 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://parquet.apache.org/docs/file-format/types/variantencoding/ says

The Decimal type contains a scale, but no precision. The implied precision of a decimal value is floor(log_10(val)) + 1.

I don't unersatnd how that maps to 1 <= P <= 9, 0 <= S <= P 🤔

+----------------------------------------+--------------------------------------------------+---------------------------------------------+
| decimal8 (10 <= P <= 18, 0 <= S <= P) | INT64, DECIMAL(P, S) | Decimal64(P, S) |
+----------------------------------------+--------------------------------------------------+---------------------------------------------+
| decimal16 (19 <= P <= 38, 0 <= S <= P) | BYTE_ARRAY / FIXED_LEN_BYTE_ARRAY, DECIMAL(P, S) | Decimal128(P, S) |
+----------------------------------------+--------------------------------------------------+---------------------------------------------+
| date | INT32, DATE | Date32 |
+----------------------------------------+--------------------------------------------------+---------------------------------------------+
| time | INT64, TIME(false, MICROS) | Time64(us) |
+----------------------------------------+--------------------------------------------------+---------------------------------------------+
| timestamptz(6) | INT64, TIMESTAMP(true, MICROS) | Timestamp(us, UTC) |
+----------------------------------------+--------------------------------------------------+---------------------------------------------+
| timestamptz(9) | INT64, TIMESTAMP(true, NANOS) | Timestamp(ns, UTC) |
+----------------------------------------+--------------------------------------------------+---------------------------------------------+
| timestampntz(6) | INT64, TIMESTAMP(false, MICROS) | Timestamp(us) |
+----------------------------------------+--------------------------------------------------+---------------------------------------------+
| timestampntz(9) | INT64, TIMESTAMP(false, NANOS) | Timestamp(ns) |
+----------------------------------------+--------------------------------------------------+---------------------------------------------+
| binary | BYTE_ARRAY | Binary / LargeBinary / BinaryView |
+----------------------------------------+--------------------------------------------------+---------------------------------------------+
| string | BYTE_ARRAY, STRING | String / LargeString / StringView |
+----------------------------------------+--------------------------------------------------+---------------------------------------------+
| uuid | FIXED_LEN_BYTE_ARRAY[len=16], UUID | :ref:`UUID extension type <uuid_extension>` |
+----------------------------------------+--------------------------------------------------+---------------------------------------------+

The decimal precision bands follow the `Variant encoding types

@alamb alamb Aug 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am still a little confused about how decimals are working -- is the this text talking about each variant value itself (as the scale/precision can vary row to row) or the min/max precision of all values in a column (presumably what is needed when shredding) 😕

<https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types>`__
table: the bands are disjoint, so precision alone selects the row (the
narrowest sufficient decimal type is required) and the scale must satisfy
``0 <= S <= P``. Arrow decimal types outside these bounds (a negative scale,
or a wider decimal type than the precision requires) are not valid
``typed_value`` storage.

.. note::

Arrow types without a row in this table (such as ``Null`` or the unsigned
integer types) must not be used as ``typed_value`` storage, as they have no
valid Parquet shredded representation:

* A Variant null is always encoded in the ``value`` field (as ``00``),
never in ``typed_value``: a null ``typed_value`` signals that the row is
not shredded, and for shredded object fields a null ``typed_value``
together with a null ``value`` means the field is missing.

* Variant has no unsigned integer types, so unsigned Arrow values must be
converted to a signed Variant type wide enough to hold them (for example,
``Uint8`` values become ``int16``) before being stored in ``value`` or in
a signed integer ``typed_value`` column.

.. _timestamp_with_offset_extension:

Expand Down
Loading