Skip to content

feat(intelligence): let a hand-authored candidate enter the profile-improvement path — propose* currently privileges improve() #751

Description

@drewstone

The contradiction

proposeAgentProfileImprovement hard-calls improve() and throws unless it returned decision: 'ship' (improvement-cycle.ts:800-811). The sibling proposeAgentImprovement does the same. So the only front door into the canonical seal → measure → compare → proposal chain requires the candidate to have been produced by Runtime's own optimizer harness.

That privileges one generator, which contradicts the improvement doctrine this machinery was built to serve: "There is no privileged improvement generator" — one canonical path for identity, sealing, execution, evidence, comparison, review, and activation, with many legal authorship routes into it (direct agent/human authorship, imports, GEPA, evolutionary mutation, recombination).

Why it matters, measured

The discovery lab's evolution round authors candidates with a trace-reading agent that emits a canonical AgentProfileDiff (applied via applyAgentProfileDiff, digest-pinned), and evaluates them with 6-hour, 10–40M-token supervised research runs graded by a blind oracle. That regime cannot run through improve(): a method-mode call structurally requires ≥4 disjoint scenarios and ~6 rollouts minimum for a shippable comparison (compare-optimization-methods.ts:858-907; the test fixture at improve.test.ts:33-39 uses exactly the structural minimum), and the repo's own doctrine warns against building the optimizer over a metric you can only sample a few times.

The open port is otherwise perfect for this: AgentProfileImprovementExecutor.measure is an arbitrary caller function, timeoutMs has no upper bound, and the receipt's grading block treats an external oracle as first-class (digest-named grader, never executed on this path). Everything downstream — sealed experiment, paired evaluation, proposal, review, activation — accepts it. The only thing a hand-authored candidate cannot do is get in the door.

Ask

A propose path (or an exported variant) that accepts a caller-supplied candidate profile + its AgentProfileDiff + authorship provenance in place of the internal improve() call, and runs the identical seal → measure → compare → proposal chain. The building blocks are already individually exported (sealAgentProfileImprovementExperiment, runAgentProfileImprovementExperiment, measuredComparisonFromAgentProfileImprovementExperiment, createAgentImprovementProposal), so today a caller can compose them by hand — but then the freshness guards, ship-gate ordering, and lineage stamping that proposeAgentProfileImprovement encodes have to be re-derived by every caller, which is exactly the drift the one-path doctrine exists to prevent.

Evidence gathered by an 8-agent source read (agent-runtime @ 7dcfca4, agent-eval 0.144.6); line cites verified by hand.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions