Skip to content

feat(scoring): wire issue-discovery/issue-spam scoring constants into engine - #835

Merged
JSONbored merged 10 commits into
JSONbored:mainfrom
YB0y:feat/issue-discovery-scoring-808
Jun 20, 2026
Merged

feat(scoring): wire issue-discovery/issue-spam scoring constants into engine#835
JSONbored merged 10 commits into
JSONbored:mainfrom
YB0y:feat/issue-discovery-scoring-808

Conversation

@YB0y

@YB0y YB0y commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Ten scoring constants were parsed and stored in ScoringModelSnapshotRecord but never applied anywhere in the scoring engine — causing silent drift between upstream gittensor and gittensory's preview outputs.

This PR resolves the drift by:

  • Wiring TEST_FILE_CONTRIBUTION_WEIGHT into computeScoreCore() so test-file tokens are weighted at 0.05× when computing totalTokenScore from components.
  • Wiring the open-issue spam gate (OPEN_ISSUE_SPAM_BASE_THRESHOLD, OPEN_ISSUE_SPAM_TOKEN_SCORE_PER_SLOT, MAX_OPEN_ISSUE_THRESHOLD) as a new scoring multiplier that mirrors the existing open-PR spam gate.
  • Formally scoping out the five upstream eligibility-floor constants. (PR_LOOKBACK_DAYS, MIN_VALID_MERGED_PRS, MIN_VALID_SOLVED_ISSUES,
    MIN_ISSUE_CREDIBILITY, MIN_TOKEN_SCORE_FOR_VALID_ISSUE) with inline comments explaining they are validator-enforced, not per-PR/issue scoring inputs.
  • Exposing the new open_issue_threshold blocker in ScoreGateBlocker, ScoreGateDelta, ScorePreviewResult, and PUBLIC_BLOCKER_TEXT for public-safe scenario summaries.

Related Issue

Closes: #808

Change Type

  • Bug fix (silent scoring drift — constants stored but never applied)
  • New feature (open-issue spam gate scoring lane)
  • Refactor (scope-out comments for unmodeled eligibility-floor constants)
  • Test (6 new unit tests covering all wired constants and gate behavior)
  • Documentation
  • Breaking change

Real Behavior Proof

TEST_FILE_CONTRIBUTION_WEIGHT wired:

input: sourceTokenScore=100, testTokenScore=200, nonCodeTokenScore=0
before: totalTokenScore = 100 + 200 + 0 = 300   ← test tokens at full weight (wrong)
after:  totalTokenScore = 100 + 0.05×200 + 0 = 110 ← test tokens at 0.05× (correct)

Open-issue spam gate blocks at threshold:

OPEN_ISSUE_SPAM_BASE_THRESHOLD=2, existingContributorTokenScore=0
→ openIssueThreshold = min(30, 2 + floor(0/300)) = 2
openIssueCount=3 → openIssueMultiplier=0 → estimatedMergedScore=0
blocker: { code: "open_issue_threshold", severity: "blocker" }

Threshold scales with token history:

existingContributorTokenScore=900
→ openIssueThreshold = min(30, 2 + floor(900/300)) = min(30, 5) = 5
openIssueCount=4 → openIssueMultiplier=1 → score unaffected

MAX_OPEN_ISSUE_THRESHOLD cap enforced:

existingContributorTokenScore=99999
→ openIssueThreshold = min(30, 2 + floor(99999/300)) = 30 (capped)

All 9 previously unmodeled constants no longer flagged:

npm run test:unit → "all 9 issue-discovery constants are no longer unmodeled drift"  ✓

Test Output

npm run typecheck   → exit 0 (0 errors)
npm run test:unit   → 139 test files | 1959 tests passed | 0 failed

New tests (describe: "issue-discovery scoring constants (#808)"):

  1. TEST_FILE_CONTRIBUTION_WEIGHT reduces test-file token contribution
  2. open-issue count above threshold zeroes the multiplier (blocker fires)
  3. threshold scales with existingContributorTokenScore
  4. default 0 issue count never triggers the gate
  5. MAX_OPEN_ISSUE_THRESHOLD caps the threshold at 30
  6. all 9 constants are no longer flagged as unmodeled drift

Checklist

  • npm run typecheck passes (0 errors)
  • npm run test:unit passes (1959/1959)
  • No breaking changes to existing ScorePreviewResult shape (new fields are additive)
  • openIssueCount input is optional (openIssueCount?: number) — defaults to 0
  • Gate mirrors the open-PR spam gate pattern exactly (consistent behavior)
  • Public blocker text added to scenario-summary.ts for safe external rendering
  • Scope-out comments explain WHY constants are not per-PR scoring inputs

… gate into scoring engine (JSONbored#808)

Co-Authored-By: YB0y <YB0y@users.noreply.github.com>
@YB0y
YB0y requested a review from JSONbored as a code owner June 17, 2026 17:06
@dosubot dosubot Bot added the size:M label Jun 17, 2026
@ghost

ghost commented Jun 17, 2026

Copy link
Copy Markdown

Note

Gittensory Gate skipped

PR closed before full evaluation. No late first comment was created.

Signal Result Evidence Action
Gate result ⚠️ Skipped #835 is no longer open. No action.

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

@ghost ghost added the gittensory:reviewed label Jun 17, 2026
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@ghost

This comment has been minimized.

@codecov

codecov Bot commented Jun 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.60%. Comparing base (2973ff7) to head (91492a4).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #835   +/-   ##
=======================================
  Coverage   96.60%   96.60%           
=======================================
  Files         109      109           
  Lines       14914    14921    +7     
  Branches     5390     5394    +4     
=======================================
+ Hits        14408    14415    +7     
  Misses        104      104           
  Partials      402      402           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@JSONbored JSONbored added the gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. label Jun 17, 2026

@JSONbored JSONbored left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good:

  • Correctly models #808 constants in scoring defaults and score-preview math.
  • Adds useful tests for test-token weighting and open-issue threshold behavior.
  • CI/test/ui/validate pass.

Bad:

  • Codecov patch/project fail.
  • Source/generated contract is stale: no OpenAPI schema/public artifact update for new score-preview fields/blocker.
  • New open-issue gate is not wired into scenario cleanup projections, so “best reasonable case” cannot clear it.

Change requests:

  • Add OpenAPI/schema + generated artifact coverage for openIssueMultiplier/openIssueThreshold/openIssueCount/open_issue_threshold.
  • Fix Codecov patch/project failure.
  • Either project open-issue cleanup in scenario previews or explicitly document/test why it is intentionally current-state only.

@JSONbored
JSONbored self-requested a review June 18, 2026 23:50

@JSONbored JSONbored left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #835 — feat(scoring): wire issue-discovery/issue-spam constants into engine [YB0y]
Action: CHANGES REQUESTED
Issue #808: PARTIALLY CI: pass

Good:

  • Models TEST_FILE_CONTRIBUTION_WEIGHT + an open-issue spam gate mirroring the open-PR gate; scopes out
    the 5 validator-enforced floors with comments; clears 9 unmodeled-drift warnings. Good tests, and the
    scenario-cleanup projection IS wired (the maintainer's "not wired" note is addressed in this head).

Flagged:

  1. #808 asks to wire modeled constants into score preview AND decision-pack AND reward-risk — only the
    score preview is wired here; decision-pack/reward-risk untouched.
  2. OpenAPI/generated contract stale for the new openIssueMultiplier/openIssueThreshold/open_issue_threshold
    fields.

@ghost

ghost commented Jun 19, 2026

Copy link
Copy Markdown

🔍 Maintainer review

Reviewed 4 file(s) — ✅ all CI green.

Suggested action:Safe to merge — both reviewers found no blocking issues.

Reviewer A · gpt-oss-120b — recommends ✅ merge
The change introduces a new open‑issue spam gate with associated constants, updates the scoring engine, UI messages, and public blocker text, and adds comprehensive tests. Types are extended consistently, defaults handle missing values safely, and no forbidden public terms are introduced. No new external calls or secret handling are added. The implementation appears correct and maintains existing behavior. All added logic is covered by unit tests, and no CI failures are reported.

Suggestions

  • Ensure any downstream consumers of ScorePreviewResult handle the new openIssue* fields gracefully.
  • Update API documentation to reflect the new open‑issue gate and related fields.
  • Consider adding a test for serialization/deserialization of the new fields if they are part of any public API payloads.

Worth double-checking

  • Potential breakage for external integrations that expect the previous shape of ScorePreviewResult without the new fields.
  • If nonNegative(undefined) does not default to 0, callers omitting openIssueCount could cause unexpected NaN behavior.

Reviewer B · nemotron-3-120b-a12b — recommends ✅ merge
The PR wires issue-discovery/issue-spam scoring constants into the engine, adding open-issue spam gating and test-file token weight. Changes are consistent, well-tested, and maintain the public/private boundary. No forbidden terms are introduced in public output.

No blocking issues spotted.

@JSONbored
JSONbored merged commit b43f01d into JSONbored:main Jun 20, 2026
14 of 15 checks passed
@github-actions github-actions Bot mentioned this pull request Jun 20, 2026
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(scoring): model (or explicitly scope out) the issue-discovery / issue-spam dimension

2 participants