Server-side idempotency middleware with pluggable storage

[bokelley]

Tracking issue for the server-side implementation helper for AdCP #2308 — idempotency_key is now required on every mutating request. The spec defines the contract; this issue is about giving Python sellers a drop-in middleware so they don't have to build the storage/atomicity layer themselves.

Note: the repo is currently described as "a python library to interact with adcp servers" (client-focused). This issue expands the scope to server-side helpers, matching the TS SDK which covers both. If it's a better fit to split into a separate adcp-server-python package, happy to re-file there.

Sibling issues:

• Python client ergonomics: Add idempotency_key auto-generation and IdempotencyConflictError #181
• TS server middleware: adcp-client#569
• Go (combined client + server): adcp-go#42

Why a middleware

Sellers have to implement four things correctly, and the atomicity bit is what trips people up:

1. Extract idempotency_key from the request
2. Look up (principal_id, idempotency_key) in a store with TTL
3. If hit: compare canonical payload hash → return cached response, OR raise IDEMPOTENCY_CONFLICT
4. If miss: execute the handler, then atomically commit side effects + cache entry. A crash between "fire webhook" and "cache response" means the retry fires the webhook again.

Proposed API

from adcp_client.server import IdempotencyStore, canonical_json_sha256

idempotency = IdempotencyStore(
    backend=PgBackend(engine),   # or RedisBackend, or MemoryBackend for tests
    ttl_seconds=86400,
    hash_fn=canonical_json_sha256,
)

# Decorator composes with FastAPI / your MCP server framework
@idempotency.wrap
async def create_media_buy(req: CreateMediaBuyRequest, ctx: Context) -> CreateMediaBuyResponse:
    # Business logic — same transaction as the cache write
    media_buy = await db.insert_media_buy(req, ctx.tx)
    await fire_webhook(media_buy)
    return media_buy

The wrapper:

• Extracts idempotency_key from the request
• Scopes lookups by ctx.principal.id (per-principal namespace — security requirement)
• On hit with matching payload hash: returns cached response
• On hit with different hash: raises IdempotencyConflictError → mapped to IDEMPOTENCY_CONFLICT
• On miss: runs handler inside a transaction, writes {key, hash, response} alongside business writes, commits together

Backends

Ship three, MVP needs the first two:

• MemoryBackend() — in-process dict with TTL sweeper, for tests and reference agents
• PgBackend(engine, schema=None, table=None) — SQLAlchemy / asyncpg, matches transaction guarantees when business data is in Postgres
• RedisBackend(client) — redis.asyncio client with TTL-keyed entries. Document the atomicity caveat (can't transact with business DB)

Capabilities integration

capabilities.adcp.idempotency = idempotency.capability()
# → { replay_ttl_seconds: 86400 }

Docs

• "Why idempotency_key and what it costs" narrative (pointing at security.mdx)
• Snippet for each backend
• Gotchas: atomicity across backend choice, per-principal scoping, TTL implications

Acceptance

• IdempotencyStore with MemoryBackend + PgBackend
• canonical_json_sha256 hash helper (stable key order, whitespace-normalized)
• IdempotencyConflictError exception mapped to IDEMPOTENCY_CONFLICT
• @wrap decorator + async support
• .capability() method returns the capabilities fragment
• Passes the idempotency compliance storyboard end-to-end

Related: adcontextprotocol/adcp#2308, adcontextprotocol/adcp#2315, #181 (client side)

[bokelley]

Spec tightened during review (adcp#2315) — flowing the updates here for the server side:

1. Canonical form is RFC 8785 JCS. The middleware's hash_fn default MUST be JCS. Exclude idempotency_key and context from the hashed payload. Avoid Python's default json.dumps — key order is stable but float precision and escape choices aren't JCS-conformant. Use a JCS library (e.g. pyjcs or equivalent).

2. Set replayed: true on cached replays, false on fresh executions. The middleware injects this into the protocol envelope automatically — handlers don't need to set it.

3. Store request_hash, not the full request. Storing only the hash is enough for conflict detection and avoids holding the original payload at rest longer than needed (reduces blast radius of a cache-table leak).

4. IDEMPOTENCY_CONFLICT response MUST NOT include the cached payload. Middleware's conflict error should expose a field json-pointer diagnostic (best-effort) but never leak the stored payload or its fingerprint — read-oracle attack surface.

5. .capability() reads from TTL automatically. Don't require the caller to double-declare the value.

[bokelley]

Spec round 2 tightened several things — flagging for scope:

1. Cache inner response, not envelope. replayed is injected on the envelope at response time, not stored. Middleware caches the handler return value; envelope builder sets replayed based on cache hit/miss.

2. Only cache task-successful responses. On any exception from the handler, no cache write. Retry re-executes.

3. Reject post-TTL keys with IDEMPOTENCY_EXPIRED rather than silent re-execution. Keep a tombstone briefly if possible.

4. Capability is required. store.capability() MUST be wired into get_adcp_capabilities — don't let a seller ship without it.

5. Drop field json-pointer from conflict errors — schema-shape leak. IdempotencyConflictError exposes only code + message.

6. Exclude from hash: idempotency_key, context, governance_context, push_notification_config.authentication.credentials. Configurable exclude list for extensions.

7. SI session scope: si_send_message uses (principal, session_id, idempotency_key). Middleware needs a scope extractor.