Repository navigation
A long transaction committed past a concurrent tag writes metadata Java rejects #4189
Description
Apache Iceberg version
0.12.0 (latest release)
Please describe the bug 🐞
Currently, a transaction's snapshot is timestamped when staged and the commit copies that time into last-updated-ms, so a commit landing in between leaves the metadata older than its own metadata-log entry. With no commit in between, or a gap under a minute, Java loads it.
import tempfile, time
import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog
d = tempfile.mkdtemp()
cat = SqlCatalog("c", uri=f"sqlite:///{d}/c.db", warehouse=f"file://{d}")
cat.create_namespace("ns")
df = pa.table({"x": pa.array([1], pa.int64())})
t = cat.create_table("ns.t", schema=df.schema)
t.append(df)
tx = t.transaction()
tx.append(df)
time.sleep(61)
cat.load_table("ns.t").manage_snapshots().create_tag(t.current_snapshot().snapshot_id, "v1").commit()
tx.commit_transaction()
m = cat.load_table("ns.t").metadata
print(round((m.last_updated_ms - m.metadata_log[-1].timestamp_ms) / 1000))
# -61, should be >= 0
# Java 1.12.0 TableMetadataParser.fromJson on that metadata file:
# IllegalArgumentException: Invalid update timestamp 1793330655020: before the latest metadata log entry timestamp 1793330716028set_properties, create_branch, update_schema and expire_snapshots in place of the tag do the same, and a later append makes Java fail with [BUG] Expected sorted previous metadata log entries.
tested on pyiceberg 0.12.0 and main
Willingness to contribute
- I can contribute a fix for this bug independently
- I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- I cannot contribute a fix for this bug at this time
Activity