Repository navigation
feat(labs): add blind model evaluations and orchestration guidance - #53
Merged
Merged
Conversation
|
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Extend the existing identity regression and retain evaluation isolation assertions. Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
The ambient GitHub shim enabled extra config reads absent from recorded provider frames. Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
…wlines Names that followed a newline in command output, diffs, or subagent reports kept their escaped line break as a word character and slipped past the redactor. Names followed by version digits (Qwen3), bare GPT, and several newer model families also survived. Redaction now runs on each string before serialization, derives family tokens from the evaluated model id, and drops Claude's subagent usage trailer. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…onment With one environment in scope, "Select that environment" had nowhere to go. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uation Moves ModelEvaluations to migration 088 after main claimed 086 and 087. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Kiro arrived from upstream without a judge entry and reports it as unsupported, like other drivers without text generation. Two tests adapt to effect 4.1 command args and crypto layers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…conciliation Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # apps/mobile/src/features/settings/ThreadSettingsSections.tsx # evals/README.md
# Conflicts: # apps/server/src/auth/RpcAuthorization.test.ts # apps/server/src/auth/RpcAuthorization.ts # apps/server/src/observability/RpcInstrumentation.ts # apps/server/src/orchestration-v2/EffectWorker.test.ts # apps/server/src/orchestration-v2/EffectWorker.ts # apps/server/src/provider/agentContextBudget.json # apps/server/src/server.ts # apps/server/src/ws.test.ts # apps/server/src/ws.ts # packages/client-runtime/package.json # packages/contracts/src/rpc.ts # packages/contracts/src/settings.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
T3 orchestration has no project-specific record of how a model handled earlier tasks. This adds optional blind evaluations and brief guidance agents can consult when choosing models and writing delegated prompts.
Behavior
orchestrator_capabilities; explicit user model choices take precedence.Review and simplification
The review fixed selected Pi thinking options, isolated Claude execution with subscription authentication, failed-tool evidence, recent verification retention, cancelled requests, and unavailable/mixed judge selections.
The cleanup removes the duplicate evaluation runtime and silent fallback, reuses the mobile model-menu builder, shares one verdict validator, and simplifies guidance serialization and the eval runner. Dry runs write prompts only. Tests now provide evaluation dependencies explicitly.
Verification
check:changedrun hit the environment's cached typecheck launcher failure; its lint finding was fixed and rechecked, and the affected typechecks were completed without caching.This remains a draft: the repository's
test-t3-appandtest-t3-mobileskills require T3 Browser/Device tools unavailable in this session. Live web/mobile verification and judge calibration remain pending. Identity filtering cannot remove every clue from writing style or task content.Implemented and reviewed with GPT-6 Astra Pro in ChatGPT.