Handle Arrow large_string data type #520

Activity

1. kevinjqliu commented on Mar 12, 2024

   

   kevinjqliu

   on Mar 12, 2024

   ContributorAuthor

   More actions

   Looks like it was added in #382 for #226

   👍React with 👍1Reacted by Honah (Jonas) J.
2. kevinjqliu commented on Mar 12, 2024

   

   kevinjqliu

   on Mar 12, 2024

   ContributorAuthor

   More actions

   On that note, should we review the pyarrow Schema to Iceberg Schema type mappings within the repository and ensure that all types that are supported in the existing parquet type -> Spark data type -> Iceberg data type conversions are supported in parquet type -> PyArrow data type -> Iceberg data type conversions?

   #226 (comment)

   ++ to @syun64 's comment
3. kevinjqliu commented on Mar 14, 2024

   

   kevinjqliu

   on Mar 14, 2024

   Last edited by kevinjqliu

   ContributorAuthor

   More actions

   I can think of 2 options.

   1. Add Arrow LargeString as an Iceberg data type. Map 1:1 with Arrow data type. The physical representation will still be backed by string.
   2. Arrow LargeString is already converted to Iceberg String type in create_table by _convert_schema_if_needed (see Arrow: Support large-string #382). So when writing an Arrow table (in overwrite/append), convert the given Arrow table schema to the table's schema, after checking the two schemas are compatible.

   Example:
   

   iceberg-python/pyiceberg/table/__init__.py

   Line 1138 in 36a505f

   _check_schema(self.schema(), other_schema=df.schema)

           _check_schema(self.schema(), other_schema=df.schema)
           # safe to cast
           from pyiceberg.io.pyarrow import schema_to_pyarrow
           pyarrow_schema = schema_to_pyarrow(self.schema())
           df = df.cast(pyarrow_schema)
   

   WIP example in #523

   @Fokko @HonahX @syun64 would love your opinions on this
4. Fokko

   closed this as completedin #523on Mar 28, 2024