perf(explorer): batch run reads and reuse entity scans

[morluto]

Closes #78.

Reuse each run-group list request's run records, group them once, and fetch only the selected page's instances in batches of at most 500 IDs. Share funnel aggregation across list/detail paths; detail and SSE polls continue to obtain fresh state.

Unfiltered entity pages now obtain latest rows and distinct totals through one concrete store operation. Generic implementations share one scan; custom pagination/count overrides and namespace routing retain their behavior.

Workload	Before	After	Speedup
1,000 runs, 100 groups, page of 20	174.21 ms	7.63 ms	22.8×
2,000 entities × five versions, page of 50	20.67 ms	11.53 ms	1.8×

Calls per request	Before	After
Run queries	63	3
Instance reads	200 individual reads	1 indexed batch lookup
Entity scans	2	1

Timings cover Python endpoint functions and temporary SQLite stores, excluding HTTP/network latency. Python 3.12.13 on macOS arm64; baseline 602d730; median of three calls. Both fixtures produce identical response hashes. Both fixtures now seed the production instance-ID index. SQLite ID batches search that index; DynamoDB uses consistent primary-key batches of at most 100 keys and retries unprocessed keys. Remote-provider latency is unmeasured.

Reproduce:

PYTHONPATH=src:packages/agentenv-protocol/src .venv/bin/python tst/benchmarks/explorer_reads.py --base 602d730

Validation:

• Full unit and protocol suites: 6,252 passed, 13 skipped. Focused Explorer/store regressions: 378 passed.
• Explorer/store tests cover pagination, large groups, selected-page reads, missing instances, tied timestamps, cross-version aggregation, fresh SSE polls, custom backend overrides and namespace merging.
• Shared conformance coverage also checks first-match ID lookup and request ordering. Regressions verify indexed SQLite reads decode only requested rows, DynamoDB never scans indexed ID batches, retries cannot silently omit keys, and routed duplicate instances retain the original precedence.
• Independent review completed with no actionable findings; plugin API check and diff checks passed.

Separate commits keep run-group batching and entity-page scan reuse reviewable. This PR is independent of the task, journal and existing context-diff PRs.

Confidence Score: 5/5

The PR appears safe to merge; the prior findings are addressed, and no new actionable issue was found.

What we checked:

• Latest-version reads keep their order: SQLite uses the indexed result only when its version is an integer. Otherwise it uses the original sorted read.
• Routed reads keep their namespace: The routed lookup filters by the entity ID. A single-ID filter selects its configured or local namespace, as it did before.

Summary

Explorer run-group lists now reuse run records and fetch instances in batches, while unfiltered entity pages share the work used to get rows and totals. These changes reduce repeated store reads for both workloads.

• Run-group lists reuse task runs and fetch selected instances in batches.
• Unfiltered entity pages get latest rows and totals from one store operation.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Run-group list] --> B[Read runs once]
  B --> C[Select group page]
  C --> D[Batch-read selected instances]
  D --> E[Build group counts and rows]
  F[Entity list] --> G[Latest-per-ID page and total]

Loading

_{Reviews (3) · Last reviewed commit: "fix(explorer): resolve conflicts with la..." · Reviewed by Greptile}

[psinghal20]

Hi @morluto! Welcome to the project, and thank you for your contribution! I looked at the PR, LGTM overall but we should address the Greptile reviews before we can merge it.

[psinghal20]

Approved, Thanks for addressing the feedback!

[earakely-scale]

Mind resolving merge conflict

[earakely-scale]

lgtm, thanks