Glue table Parameters are never pruned: removed table properties persist indefinitely

[sungwy]

_construct_parameters in pyiceberg/catalog/glue.py builds the Glue Parameters map by starting from whatever Glue already holds and layering the current table properties on top:

new_parameters = glue_table.get("Parameters", {}) if glue_table else {}
new_parameters.update({TABLE_TYPE: ICEBERG.upper(), METADATA_LOCATION: metadata_location})
if prev_metadata_location:
    new_parameters[PREVIOUS_METADATA_LOCATION] = prev_metadata_location

if metadata_properties:
    for key, value in metadata_properties.items():
        new_parameters[key] = str(value)

The operation is additive only. A key that exists in Glue but is no longer present in the table's metadata is never deleted, so removing a property from the table and committing leaves the old value in Glue permanently.

Glue Parameters and the table's property map therefore drift apart over time, and the drift is one-directional: Glue accumulates every key the table has ever had. Anything that reads table properties via Glue sees keys the table no longer declares.

---

Issue investigation generated via claude, reviewed by Sung, Kevin, Fokko.

[ArulJerald]

I'd like to pick it up and raise the PR.

@sungwy - Could you please clarify ?

While working on this I hit a design question worth settling before opening a PR.

The straightforward fix is to rebuild Parameters from the reserved keys (table_type, metadata_location, previous_metadata_location) plus the current table properties, rather than layering onto whatever Glue already holds. That prunes removed properties correctly.

It also drops keys PyIceberg never wrote, though. Injecting externally-managed parameters into a moto-backed Glue table and then running an ordinary property commit:

key	today	with rebuild
classification	preserved	deleted
UPDATED_BY_CRAWLER	preserved	deleted
recordCount	preserved	deleted

So parameters written by Athena or a Glue crawler would disappear on the next commit.

The alternative is to prune surgically — remove only keys that were in the previous table properties and are absent from the new ones, leaving anything else alone. Both current_glue_table and current_table.properties are already in scope at the commit_table call site, so it stays small:

new_parameters = dict(glue_table.get("Parameters", {})) if glue_table else {}
for key in set(prev_metadata_properties) - set(metadata_properties or {}):
    new_parameters.pop(key, None)

Which behavior do you want?

1. Full rebuild — Glue Parameters becomes an exact mirror of the Iceberg table properties; anything else isn't PyIceberg's to keep.
2. Surgical prune — only properties the table itself dropped are removed; externally managed keys survive.

Happy either way. (2) seemed the safer default, but (1) may well be intended if Parameters is meant to mirror table properties exactly.