Python: perf(foundry-hosting): cache FoundryStateStore in FoundryAgentSessionStore - #8281
Open
Harsheet Shah (harsheet-shah) wants to merge 5 commits into
Open
Conversation
…Store Reuse one process-wide FoundryStateStore for agent-session persistence instead of rebuilding it (new credential + agent_sessions metadata round-trip via get_or_create) on every get/set/delete. The session set() runs on the critical path of every Responses request, so the redundant work was pure per-request latency. Per-request user isolation is preserved via the per-operation call_id, so a shared store is equivalent; the store is no longer entered as an async-with context (its aclose() would defeat the cache) and is kept open for the process lifetime.
Add an autouse fixture that resets the new process-wide FoundryStateStore cache between tests so each agent-session test observes its own patched get_or_create, and add a test asserting the store is resolved once and reused across set/get/delete.
… loop and scope Address review: the store owns a loop-bound async pipeline + credential, so a process-wide singleton could be reused from a different event loop (across asyncio.run() calls or loop-scoped tests). Cache the store in a WeakKeyDictionary keyed by the running loop (closed loops -> their stores are GC'd) and by scope, with a per-loop lock, so a subclass overriding DEFAULT_ROOT_SCOPE no longer shares or clobbers the base collection. The single-loop server still shares one store, preserving the latency win.
…d subclass-scope isolation Reset the per-(loop, scope) cache between tests, and add tests asserting the backing store is resolved once under concurrent first-use and that a subclass overriding DEFAULT_ROOT_SCOPE gets its own cached store.
Harsheet Shah (harsheet-shah)
deployed
to
github-app-auth
September 11, 2026 04:58 — with 
GitHub Actions
Active
GitHub Actions
Active
Harsheet Shah (harsheet-shah)
deployed
to
github-app-auth
September 11, 2026 04:58 — with
GitHub Actions
Active
GitHub Actions
Active
Harsheet Shah (harsheet-shah)
deployed
to
github-app-auth
September 11, 2026 04:59 — with
GitHub Actions
Active
GitHub Actions
Active
Copilot started reviewing on behalf of
Harsheet Shah (harsheet-shah)
September 11, 2026 04:59
View session
agent-framework-automation
Bot
added
the
python
Harsheet Shah (harsheet-shah)
deployed
to
github-app-auth
September 11, 2026 04:59 — with
GitHub Actions
Active
GitHub Actions
Active
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Contended locks retain closed event loops, preventing cached stores and resources from being reclaimed.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Caches Foundry agent-session state stores to remove repeated credential creation and metadata requests.
Changes:
- Adds per-loop, per-scope caching with initialization locking.
- Adds cache reuse, concurrency, and scope-isolation tests.
File summaries
| File | Description |
|---|---|
_state_store.py |
Implements cached store reuse. |
test_state_store.py |
Tests caching and initialization. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 2
- Review effort level: Balanced
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
…per-loop leak Address PR review feedback on the FoundryAgentSessionStore cache: - Store the per-loop FoundryStateStore cache (and its creation lock) on the running event loop itself instead of in process-global WeakKeyDictionaries. An asyncio.Lock (and the store's pooled pipeline/credential) strongly references its loop, so a module-global map keyed by the loop kept that "weak" key alive, leaking one cache + open pipeline/credential per closed loop (e.g. every asyncio.run). Anchored to the loop, the state is reclaimed with the loop. Falls back to an uncached resolve if a C-level loop forbids attribute assignment (no reuse, but no leak). - Make the concurrent-init test's mocked get_or_create actually suspend (await asyncio.sleep(0)) so all gathered tasks contend on the creation lock; the previous non-suspending mock let the first task populate the cache synchronously, so even a lock-less implementation would have passed. - Drop the now-unnecessary global cache-reset fixture: per-loop state is isolated automatically by the function-scoped event loop each test runs on. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3d900395-13d0-4698-bf9d-f6670f9e545c
Harsheet Shah (harsheet-shah)
deployed
to
github-app-auth
September 11, 2026 09:04 — with
GitHub Actions
Active
GitHub Actions
Active
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation & Context
FoundryAgentSessionStore(the default agent-sessionSessionStorefor hosted MAF agents) rebuilds its backingFoundryStateStoreon everyget/set/delete. Each_get_store()call goes throughFoundryStateStore.get_or_create("agent_sessions", user_isolation=True), which (1) constructs a fresh credential (empty token cache → a new managed-identity token fetch) and (2) issues anagent_sessionsmetadata round-trip (GET/POST state_stores) before the actual item operation. Because the hosting infra callsset()in thefinallyof every Responses request, this redundant credential + metadata work lands on the critical path of every request.Description & Review Guide
What are the major changes?
Cache one per-event-loop, per-scope
FoundryStateStorefor the agent-session scope. The cache is aWeakKeyDictionarykeyed by the running event loop (a closed loop's store is GC'd, so a stale store is never reused on another loop), further keyed by scope (a subclass overridingDEFAULT_ROOT_SCOPEgets its own store instead of clobbering the base collection). Concurrent first-use is guarded by a per-loop lock with double-checked init. The shared store is intentionally not entered viaasync with(its__aexit__wouldaclose()the pooled pipeline + credential and defeat the cache); it is opened once and kept open for the process lifetime.What is the impact of these changes?
Only the real item
GET/PUT/DELETEremains on the hot path. Itemget/set/deletesemantics are byte-for-byte identical;get_or_createstill creates-on-first-use exactly once. Checkpoint / function-approval stores are left untouched (not on the per-request hot path). No public API change.Measured (500 cold + 500 warm streaming req/agent, concurrency 50, private/VNet Foundry project,
gpt-4o-mini, 0 errors): WARM TTLB p50 7,736 → 7,464 ms (−272 ms, −3.5%); WARM TTFB p50 flat (~5.68 s model first-token floor, unaffected). The win is elimination of the per-request credential + metadata round-trip.What do you want reviewers to focus on?
Event-loop scoping / GC of the cache, and the subclass-scope isolation.
Related Issue
Closes #8280
Contribution Checklist