A stale update_spec adds a partition field named like a column another writer added, which Java cannot load #4198

Apache Iceberg version

0.12.0 (latest release)

Please describe the bug 🐞

Currently, UpdateSpec runs the name check #2305 added against the schema it loaded, and its commit asserts only the last partition id and default spec id, so after another writer adds a column xb, a stale handle can add a partition field also named xb. Without the concurrent change the same call raises ValueError: Cannot create partition with a name that exists in schema: xb. Java then refuses the metadata.

import tempfile
import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.transforms import BucketTransform
from pyiceberg.types import LongType

cat = SqlCatalog("c", uri="sqlite:///:memory:", warehouse=f"file://{tempfile.mkdtemp()}")
cat.create_namespace("ns")
t = cat.create_table("ns.t", schema=pa.schema([("x", pa.int64())]))

with cat.load_table("ns.t").update_schema() as u:  # another writer
    u.add_column("xb", LongType())
with t.update_spec() as u:
    u.add_field("x", BucketTransform(4), "xb")

t = cat.load_table("ns.t")
print(t.schema().column_names, [f.name for f in t.spec().fields])
# ['x', 'xb'] ['xb'], should raise

Loading this metadata with Java's TableMetadataParser fails with IllegalArgumentException: Cannot create identity partition sourced from different field in schema: xb. The opposite order does the same: a stale update_schema().add_column("xb") after another writer added partition field xb commits; #4190 is another stale update_schema that AssertCurrentSchemaId lets through.

tested on pyiceberg 0.12.0 and main

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time