Repository navigation
Saving Polars Dataframe as Parquet to Iceberg #226
Description
Activity
Thank you for raising this @asheeshgarg . To me, the issue seems to be that some of the "large" types are not yet accounted for within PyIceberg type conversions.
https://arrow.apache.org/docs/python/generated/pyarrow.types.is_string.html#pyarrow.types.is_string
pa.is_string(pa.large_string()) == Falsehttps://arrow.apache.org/docs/python/generated/pyarrow.types.is_large_string.html#pyarrow.types.is_large_string
pa.is_large_string(pa.large_string()) == TrueOn that note, should we review the pyarrow Schema to Iceberg Schema type mappings within the repository and ensure that all types that are supported in the existing parquet type -> Spark data type -> Iceberg data type conversions are supported in parquet type -> PyArrow data type -> Iceberg data type conversions?
https://github.com/apache/iceberg-python/blob/main/pyiceberg/io/pyarrow.py#L849
@Fokko - should we rename this issue to "large pyarrow type support"?
Feature Request / Improvement
@Fokko
I have a polars dataframe I have save it to parquet file using the and using the Java API
DataFile dataFile = DataFiles.builder(table.spec()).withPartition(partitionKey).withInputFile(inputFile).withFormat("parquet").withRecordCount(recordCount).build();
When reading the data back using pyiceberg gettng following error
File /usr/lib64/python3.9/functools.py:888, in singledispatch..wrapper(*args, **kw)
884 if not args:
885 raise TypeError(f'{funcname} requires at least '
886 '1 positional argument')
--> 888 return dispatch(args[0].class)(*args, **kw)
File ~/code/iceberg-python/pyiceberg/io/pyarrow.py:695, in _(obj, visitor)
693 if pa.types.is_nested(obj):
694 raise TypeError(f"Expected primitive type, got: {type(obj)}")
--> 695 return visitor.primitive(obj)
File ~/code/iceberg-python/pyiceberg/io/pyarrow.py:808, in _ConvertToIceberg.primitive(self, primitive)
805 primitive = cast(pa.FixedSizeBinaryType, primitive)
806 return FixedType(primitive.byte_width)
--> 808 raise TypeError(f"Unsupported type: {primitive}")
TypeError: Unsupported type: large_string
This is the schema of the table
Schema(NestedField(field_id=1, name='id', field_type=IntegerType(), required=False), NestedField(field_id=2, name='name', field_type=StringType(), required=False), NestedField(field_id=3, name='dept', field_type=IntegerType(), required=False), schema_id=0, identifier_field_ids=[])
Same read work fine for the Spark. Do i need to use some utils to convert Polars datatype to be compatible to Pyiceberg?