Skip to content

Add blocked operational-value study for copilot-pr-merged-report - #58538

Closed
mnkiefer with Copilot wants to merge 2 commits into
mainfrom
copilot/operational-value-study-paper-v1-again
Closed

mnkiefer with Copilot wants to merge 2 commits into
mainfrom
copilot/operational-value-study-paper-v1-again

Conversation

Copilot AI commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Per .github/skills/operational-value-designer/SKILL.md, this PR runs a fresh operational-value design study for the copilot-pr-merged-report workflow and records the result — no grader is wired up because none is defensible.

Findings

  • Recovered adoption-time intent from PR [WIP] Add daily report generation for copilot pull requests #5569 (commit 0d53b1fcfa831bbe0da078b9729b33bbb3221064): the workflow only reads already-merged Copilot PR data and publishes a summary discussion.
  • The workflow never creates, resolves, or owns a repository opportunity — by the time it runs, the PRs it reports on are already merged and closed.
  • Every candidate signal (report published, PR/line/test counts, report accuracy vs. re-derived data, discussion engagement) reduces to activity, output volume, or agent-judgment — all explicitly excluded as substitutes for direct attainment.

Changes

  • Added .github/graders/copilot-pr-merged-report-operational-value-study.json with designOutcome: "blocked", including:
    • a detailed rejectionReason explaining why no direct primary metric in [0,1] can be constructed
    • classification of the workflow's domain/task/evidence characteristics
    • decisions covering intent, opportunity, evidence, and metric considerations with rejected alternatives and rationale
    • uncertainties capturing domain-fit, causal-attribution, and assignment ambiguity risks
  • No changes to .github/workflows/copilot-pr-merged-report.md, its lock file, or a grader script — per the blocked-design rule, only the study record is added.
Original prompt

[operational-value-study:paper-v1] copilot-pr-merged-report

Use the repository's .github/skills/operational-value-designer/SKILL.md to design
and verify one deterministic operational-value grader for github/gh-aw
workflow copilot-pr-merged-report. This is a fresh research run; do not copy another
workflow's metric or contract.

Requirements:

  • Recover adoption-time intent and pre-adoption evidence independently.
  • Use exactly one direct primary attainment metric in [0,1]. Keep diagnostics separate.
  • Treat unavailable evidence as null, never zero. Make maturation explicit and stable.
  • Support historical assignment reconstruction when case and event are null.
  • If no direct operational metric is defensible, create only the study record below with designOutcome "blocked" and explain why. Never substitute activity, output volume, traces, or agent judgment.
  • Change only .github/workflows/copilot-pr-merged-report.md, its generated lock file,
    .github/graders/copilot-pr-merged-report-operational-value.sh, and .github/graders/copilot-pr-merged-report-operational-value-study.json.
  • Do not add generated historical reports to the pull request.

Write .github/graders/copilot-pr-merged-report-operational-value-study.json as JSON with this exact top-level shape:
{
"schemaVersion": 1,
"repository": "github/gh-aw",
"workflowId": "copilot-pr-merged-report",
"recordedAt": "ISO-8601 timestamp",
"recorder": "copilot-coding-agent",
"designOutcome": "accepted|rejected|blocked",
"rejectionReason": null,
"classification": {
"domain": "short research domain",
"subdomain": null,
"taskArchetype": "short label",
"interventionType": "short label",
"outcomeType": "short label",
"evidenceDirectness": "direct|proxy|activity",
"evidenceLocality": "same-repository|cross-repository|external",
"measurementLevel": "nominal|ordinal|interval|ratio",
"metricForm": "binary|proportion|rate|distance-to-target|index",
"assignmentDeterminism": "deterministic|bounded-ambiguity|ambiguous",
"intentExplicitness": "explicit|implicit|mixed",
"markdownSpecificity": "exact|bounded|open-ended",
"outcomeObservability": "immediate|delayed",
"causalClaim": "observational|comparative-not-causal"
},
"decisions": [{
"type": "intent|opportunity|assignment|evidence|maturation|metric|baseline|diagnostic",
"selected": "observable selected design",
"alternatives": ["observable rejected alternative"],
"rationale": "concise evidence-backed rationale, not hidden reasoning",
"evidenceReferences": ["commit, issue, PR, or file reference"],
"changedDuringReview": false
}],
"uncertainties": [{
"type": "measurement|assignment|evidence-availability|temporal|baseline|domain-fit|causal-attribution",
"level": "low|medium|high|unknown",
"description": "specific limitation",
"mitigation": null,
"evidenceReferences": []
}]
}

For blocked/rejected designs, rejectionReason must be a non-empty string. Record
observable contract choices and alternatives only; do not record chain-of-thought.

Validation for accepted designs:

  1. .github/skills/operational-value-designer/scripts/verify-operational-value-evaluator.sh .github/graders/copilot-pr-merged-report-operational-value.sh
  2. gh aw compile .github/workflows/copilot-pr-merged-report.md

Keep the pull request limited to this workflow. The local study runner will replay
history and archive reports after this task completes.

Co-authored-by: mnkiefer <8320933+mnkiefer@users.noreply.github.com>
Copilot AI changed the title [WIP] Design and verify operational-value grader for copilot-pr-merged-report Add blocked operational-value study for copilot-pr-merged-report Sep 4, 2026
Copilot AI requested a review from mnkiefer September 4, 2026 15:15
@mnkiefer mnkiefer closed this Sep 4, 2026
@mnkiefer
mnkiefer deleted the copilot/operational-value-study-paper-v1-again branch September 4, 2026 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants