Skip to content

chore(agents): add a PR creation and maintenance skill - #9926

Open
saphid wants to merge 16 commits into
pingdotgg:mainfrom
saphid:chore/pr-evidence-skill
Open

chore(agents): add a PR creation and maintenance skill#9926
saphid wants to merge 16 commits into
pingdotgg:mainfrom
saphid:chore/pr-evidence-skill

Conversation

@saphid

@saphid saphid commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

The skill now omits animated GIFs when the changed behavior is nonvisual. Accessible-name and ARIA fixes use direct before/after semantic observations; visible changes and motion retain the GIF requirement. This clarification is documentation-only (23161a66c); skill validation and git diff --check passed. Existing test and behavioral-eval results below apply to their recorded revisions.

Agents preparing a PR can stop at a description or local screenshots without publishing the reviewable result. This adds prepare-pr, a skill that owns the complete PR workflow: branch selection, focused verification, commit, push, PR creation or update, required visual evidence, publication, readiness, and final readback. AGENTS.md routes full PR work—including draft completion and visual proof—to this skill, while artifact-only requests use its supporting media branch.
Fresh capture now begins by selecting a same-surface recorder with concrete start, stop, and file-export operations, then proving it with a saved, decoded, visually inspected smoke clip before a long verification run. The workflow reuses that proven route for comparable base/candidate captures, inspects each export immediately, and retains a receipt for resuming without rebuilding a working setup. Recovery is a fallback after capture setup fails, rather than the primary workflow. Existing authorization persists through capture and publication.

The workflow treats a published draft as an intermediate checkpoint when the user asked to complete a PR. It inventories authorization once, continues without redundant confirmation for actions already authorized, and asks once at the concrete client-automation boundary when repository policy still requires consent. Changes with a visible before/after difference require embedded animated GIFs showing labeled base and candidate states. Nonvisual accessibility changes omit GIFs and use observed accessibility-tree or DOM differences; different captions over unchanged UI do not count as visible proof. Every claimed motion or interaction requires a recording-derived GIF, with real-time recordings retained for timing proof. Evidence stacks vertically: side-by-side tables, columns, and composite comparisons are prohibited. The final review checks the PR content at a 390 CSS-pixel mobile viewport.

Behavioral evals before the GIF-policy revision

The disposable fixture uses real local Git remotes plus simulated GitHub attachment and client-capture services. Four existing scenarios continue to cover PR creation, existing-PR updates, description-only scope, and upload failure. This revision adds the reported failure mode directly:

Scenario Observed result
Create a PR Intended change committed and pushed; one draft PR created; both supplied files uploaded, linked, fetched, and read back.
Update an existing PR All three screenshots and two videos published and retrieved; existing head preserved; missing baseline and unusable media remain explicit draft holds.
Description only Proposed title/body delivered; Git, published PR, and attachments unchanged.
Upload failure Authorized upload attempted; actual service error reported; source preserved; no invented URL or authorization blocker.
Authorized animation PR A fresh agent used the already-authorized client path without asking again, captured and inspected base/candidate screenshots and real-time recordings, updated the existing draft, uploaded and fetched all four artifacts, marked it ready, and read back the final state.

The new animation scenario was run twice in fresh fixtures. Its first independent semantic review rejected readiness because the fixture's initial MP4s decoded to only one unique frame: attachment hashes passed, but no motion was proven. After replacing them with genuine ten-frame stepped and continuous recordings, the rerun passed observable checks and independent semantic review. The reviewer decoded the paths as:

  • Base: 10, 10, 10, 60, 60, 60, 110, 110, 110, 110
  • Candidate: 10, 22, 34, 46, 58, 70, 82, 94, 106, 118

This demonstrates that fixture checks alone cannot establish skill behavior or media quality. The evaluator instructions require a separate review of the original request, frozen inputs, Git/PR/attachment state, media, and handoff. These are offline workflow evals, not proof of real GitHub authentication, Markdown rendering, or live T3 UI behavior.

Verification

Capture reliability revision a32565fcc: 15 existing fixture tests passed; relative-link checks, installed/source capture-reference equality, and git diff --check passed. A fresh GPT-5.6 Sol high agent evaluated eight frozen offline cases, including saved-file smoke verification before a long run and reuse of an unchanged proven recorder. Primary GPT-6 review accepted all eight decisions with no actionable findings. These are offline decisions, not a live recorder integration test or a claim that QuickTime recording is fixed. Skill schema validation could not run in this shell because Python lacks PyYAML; the frontmatter is unchanged.

A fresh direct review launch (claude --model claude-opus-5 --effort high --print --tools '' --no-session-persistence --output-format json) exited 1: OAuth expired before model execution; no Claude review occurred. Current-head CI is pending at publication.

GIF-policy and evaluator revision 674c0f4fe: 32 focused tests passed (15 eval fixture, 17 media helper); skill validation and git diff --check passed. A fresh GPT-5.6 Sol animation scenario produced and published separately labeled base/candidate GIFs plus MP4s and PNGs. Parent review decoded both ten-frame GIFs, inspected their motion, and verified all six attachment hashes and vertical embedding. The new GIF/layout requirements passed this content check. The initial run exposed evaluator defects: derived GIF uploads were rejected, and gh pr ready --help changed readiness. Both are fixed with focused regression coverage. The second fresh GPT-5.6 Sol run passed the observable checker (exit 0) and independent parent semantic review with no actionable findings: two labeled GIFs embed vertically, all six artifacts match their uploaded hashes, and capture/upload/fetch/edit/ready/readback ordering is correct. The failed first receipt is retained. No live-client behavior is claimed. Direct Claude Opus 5 high review was retried and exited 1 because OAuth expired, before any model execution.

Head 674c0f4fe merges cleanly with fetched upstream main 6270a6f88bea4c2fe07a43e69693a918cf94a353 (git merge-tree --write-tree, exit 0).

  • Fresh animation_pr agent run: observable checker passed; independent semantic review passed with no actionable findings after decoding the recordings and reconciling Git, attachments, readiness, final readback, limitations, and frozen hashes.
  • python3 -B -m unittest discover -s .agents/skills/prepare-pr/evals -p 'test_*.py': 15 passed.
  • python3 -B -m unittest discover -s .agents/skills/prepare-pr/scripts -p 'test_*.py': 17 passed.
  • Skill validation and git diff --check passed.
  • CI at 524ab1e15 found one Markdown table-formatting issue in the new eval README; ebf6b4ca4 repairs it. Checks on ebf6b4ca4 passed: Check, Test, all three server shards, Rust, Release Smoke, and Mobile Native Changes. All reported checks on final head 674c0f4fe passed or were skipped. Scoped vp fmt ... --check remained unavailable locally because this worktree has no project-local vite-plus installation.
  • Direct cross-provider review command claude --model claude-opus-5 --effort high --print ... exited 1 before model execution because the OAuth session was expired. No Claude/Opus review occurred.

The evaluator accepts GIF derivatives while preserving uploaded-byte integrity and original-capture checks; independent semantic review remains responsible for verifying actual animation, baseline fidelity, and vertical presentation. Readiness help is now read-only.

The Python/ImageMagick helper crops PNGs and GIFs while retaining comparable framing, frames, timing, and loop settings. Its executable bytes are unchanged by this completion update, so the existing synthetic crop examples below remain applicable.

Crop examples and evidence limits

The same synthetic source is shown whole and cropped. The detail retains the complete task label, status, and surrounding context.

Full source, 1440×900

Synthetic source: the task status occupies a small part of the canvas

Contextual detail, 960×287

Same source cropped to retain the complete task label and status

Animated before/after status example

Before: working; after: completed. Synthetic still-state comparison.

This GIF alternates labeled synthetic baseline and candidate stills, two seconds per state. It illustrates the static comparison format; it is not motion or live-app proof.

The baseline status crop uses the same rectangle and scale.

Synthetic moving control: a fixed crop retains all four positions

The GIF retains its four synthetic frames and original 70/230/110/500 ms delays. These examples demonstrate the helper's output; they do not verify live T3 behavior or application responsiveness.

Implemented with GPT-6 and GPT-5.6 Sol in Codex/T3. The completion audit and fixture repair used GPT-5.6 Sol; independent behavioral execution and semantic review used the GPT-5 family through the Codex harness. No Claude/Opus reviewer participated.

Current-base verification

Updated to upstream main b2e15185aee22bfdee78806f2734b0a4de9c2c77 at 19c752242c556dd4900df15251109595fc887fd3. All 3 applicable focused verification commands passed (32 passing test executions). Earlier runtime observations and media retain their stated revisions and limitations; these fresh runs do not constitute new client recordings. Fresh independent Claude review is unavailable because its OAuth credentials have expired. Refreshed by GPT-6 Astra in the Codex harness.

Note

Add prepare-pr skill for PR creation, maintenance, and visual evidence

  • Adds the prepare-pr skill definition in SKILL.md covering end-to-end PR workflow: request scoping, branch inspection, behavior verification, visual evidence capture, PR description, publication, and final readback
  • Adds media scripts prepare_proof_media.py and detail_crop.py for creating contextual PNG/GIF detail crops with transactional packet publication and rollback
  • Adds a behavioral eval fixture in fixture.py with simulated gh and ui-proof wrappers, plus invariant tests in test_fixture.py covering 18 scenarios including upload failures, tampered evidence, and concurrent operations
  • Adds reference docs for capture recovery, detail crops, media workflow, and PR examples; updates AGENTS.md to designate prepare-pr as the owner of PR creation and maintenance
  • Risk: AGENTS.md now routes all PR creation and updates through the prepare-pr skill; existing PR workflows that bypass this skill may need adjustment

Macroscope summarized 19c7522.

@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XL 500-999 changed lines (additions + deletions). labels Sep 5, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 91567f2. Configure here.

Comment thread .agents/skills/prepare-pr/scripts/detail_crop.py
@macroscopeapp

macroscopeapp Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at a32565f

Macroscope's review found this PR approvable — This PR adds a substantial prepare-pr workflow and supporting evaluation/media tooling, but all changes are confined to development-agent operations and local harnesses. It does not alter the shipped product, production infrastructure, product defaults, or static-analysis configuration.

Notes:

  • No code objects were reviewed. Approvability was decided on eligibility alone.

You can add or adjust custom eligibility rules. Learn more.

@github-actions github-actions Bot added size:XXL 1,000+ changed lines (additions + deletions). and removed size:XL 500-999 changed lines (additions + deletions). labels Sep 5, 2026
macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Sep 5, 2026
@macroscopeapp
macroscopeapp Bot dismissed their stale review September 5, 2026 05:50

Dismissing prior approval to re-evaluate 6d295b6

@saphid saphid changed the title chore(agents): prepare PR descriptions and readable visual proof chore(agents): add a PR preparation and maintenance skill Sep 5, 2026
macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Sep 5, 2026
macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Sep 5, 2026
@macroscopeapp
macroscopeapp Bot dismissed their stale review September 5, 2026 10:12

Dismissing prior approval to re-evaluate 9984d3b

@saphid saphid changed the title chore(agents): add a PR preparation and maintenance skill chore(agents): add a PR creation and maintenance skill Sep 5, 2026
macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Sep 5, 2026
@cursor

cursor Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

@macroscopeapp
macroscopeapp Bot dismissed their stale review September 5, 2026 14:00

Dismissing prior approval to re-evaluate 524ab1e

macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Sep 5, 2026
@macroscopeapp
macroscopeapp Bot dismissed their stale review September 5, 2026 14:06

Dismissing prior approval to re-evaluate ebf6b4c

macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Sep 5, 2026
@macroscopeapp
macroscopeapp Bot dismissed their stale review September 5, 2026 23:04

Dismissing prior approval to re-evaluate a7b9f90

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL 1,000+ changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant