Skip to content

feat(miner): add local portfolio/queue store - #2751

Merged
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
dhgoal:feat/miner-portfolio-queue-store
Jul 3, 2026
Merged

feat(miner): add local portfolio/queue store#2751
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
dhgoal:feat/miner-portfolio-queue-store

Conversation

@dhgoal

@dhgoal dhgoal commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds the local portfolio/queue store to @jsonbored/gittensory-miner — the last piece of client-side persistence the foundation phase calls for. It tracks the miner's own backlog of candidate work items across every repo it has been pointed at ("what should I look at next, across everything I'm tracking"), backed by a small local SQLite table. The store never uploads, syncs, or phones home — it only lives on the miner's machine, exactly like the existing run-state store it mirrors (lib/run-state.js, #2289).

Scope is persistence + a FIFO/priority-ordered read-write API only. The priority field is a placeholder numeric input in this phase; later phases populate it from the extracted reward-risk/scoring modules in gittensory-engine — it is not invented here.

API (lib/portfolio-queue.js):

  • enqueue({ repoFullName, identifier, priority? })QueueEntry. Re-enqueueing a tracked item re-activates it in place: it refreshes the (placeholder) priority and resets status to queued, but keeps the original enqueued_at/rowid so it holds its existing FIFO position instead of jumping the queue (restamping would be inconsistent — the fixed rowid still pins the old position when timestamps collide — so position is deliberately preserved).
  • dequeueNext()QueueEntry | null — highest-priority-first, claims the item as in_progress; null on an empty/all-claimed queue.
  • listQueue(repoFullName?) — all items, or one repo's, in the same order.
  • markDone(repoFullName, identifier) — transitions to done (excluded from future dequeueNext); null for a missing item.

Ordering is priority DESC, enqueued_at ASC, rowid ASC: the implicit rowid guarantees the insertion-order tie-break even when two items share both a priority and an enqueued_at timestamp (the test freezes the clock to prove exactly that, rather than relying on timestamp skew).

Schema is per the issue spec: PRIMARY KEY (repo_full_name, identifier), priority REAL NOT NULL DEFAULT 0, status TEXT NOT NULL DEFAULT 'queued' CHECK(status IN ('queued','in_progress','done')), enqueued_at TEXT NOT NULL.

Closes #2292.

Scope

Validation

  • git diff --check
  • npm run typecheck
  • npm run test:coverage locally — the new test/unit/miner-portfolio-queue.test.ts passes (all 10 miner test files, 110 tests, green). This change lives entirely in packages/**, which Codecov does not measure, so it carries no codecov/patch obligation; the logic is nonetheless exercised on both sides of every branch (empty vs non-empty dequeue, present vs missing markDone, default vs explicit priority, single- vs multi-repo listing, re-enqueue upsert, malformed-input rejection).
  • node --check lib/portfolio-queue.js via npm run --workspace @jsonbored/gittensory-miner build
  • npm audit --audit-level=moderate — this PR adds no dependencies, so dependency-review has nothing new to evaluate.
  • New behavior has unit tests for new branches, fallback paths, and ordering invariants.

If any required check was skipped, explain why:

  • UI/OpenAPI/migration/workers checks are not applicable: this change is one local-persistence module in packages/gittensory-miner/lib plus its test — no src/**, UI, API schema, Drizzle, or Cloudflare-binding surface is touched. The SQLite table is a miner-local file, not a repo migration.

Safety

  • No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed. The DB is created 0o600 in a 0o700 dir, owner-only, and never leaves the machine.
  • Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
  • Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests. — n/a: local-only SQLite persistence, no auth/session/network surface.
  • API/OpenAPI/MCP behavior is updated and tested where needed. — n/a: no API/OpenAPI/MCP surface changed.
  • UI changes use live API data or real states. — n/a: no UI change.
  • Visible UI changes include a UI Evidence section. — n/a: no visible UI, frontend, docs, or extension change.
  • Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs.

Notes

Additive and consistent with the package's existing local-store pattern: two new files (lib/portfolio-queue.js + its lib/portfolio-queue.d.ts) mirroring lib/run-state.js / lib/run-state.d.ts (path resolution, 0o600/0o700 perms, prepared statements, default-store singleton), one line added to the package build (node --check) gate, and one new test file mirroring test/unit/miner-run-state.test.ts. No existing code is modified.

@dhgoal
dhgoal requested a review from JSONbored as a code owner July 3, 2026 11:28
@superagent-security superagent-security Bot added the contributor:flagged Contributor flagged for review by trust analysis. label Jul 3, 2026
@superagent-security

Copy link
Copy Markdown
Contributor

🚨 Contributor flagged. Click here for more info: Superagent Dashboard

@dhgoal

dhgoal commented Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

Fixed the re-enqueue ordering inconsistency. Re-enqueueing a tracked item now re-activates it in place — it refreshes the (placeholder) priority and resets status to queued, but keeps the original enqueued_at and rowid, so it holds its existing FIFO position instead of claiming a reposition that the fixed rowid would contradict when timestamps collide. rowid is now the stable, unique total-order tie-break. Added a regression test that freezes the clock so two items share an enqueued_at, then re-enqueues one and asserts the order is unchanged.

@codecov

codecov Bot commented Jul 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.13%. Comparing base (2a522e4) to head (601a172).
⚠️ Report is 10 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2751      +/-   ##
==========================================
+ Coverage   96.12%   96.13%   +0.01%     
==========================================
  Files         248      248              
  Lines       27548    27569      +21     
  Branches    10007    10012       +5     
==========================================
+ Hits        26480    26503      +23     
  Misses        443      443              
+ Partials      625      623       -2     

see 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@loopover-orb loopover-orb Bot added the gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. label Jul 3, 2026
@loopover-orb

loopover-orb Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ Gittensory review result - manual review recommended

Review updated: 2026-07-03 19:13:58 UTC

4 files · 1 AI reviewer · no blockers · readiness 80/100 · CI green · clean

⏸️ Suggested Action - Manual Review

Review summary
This adds a local SQLite-backed portfolio queue with path resolution, validation, priority/FIFO ordering, default singleton helpers, and focused unit coverage for the main API contract. The core queue operations are coherent: re-enqueue updates in place, `dequeueNext()` claims with a single `UPDATE ... RETURNING`, and the `.d.ts` surface matches the implementation. I do not see a reachable correctness defect in the provided diff; the remaining issues are maintainability and defensive-hardening details around the new local store.

Nits — 5 non-blocking
  • nit: `packages/gittensory-miner/lib/portfolio-queue.js:78` only applies `0o700` to newly-created directories, so an existing `GITTENSORY_MINER_CONFIG_DIR` with broad permissions remains broad even though the DB file is owner-only.
  • nit: `packages/gittensory-miner/lib/portfolio-queue.js:132` lets `close()` be called repeatedly and will surface the underlying SQLite close error on the second call; make the store close operation idempotent if callers may use cleanup-style `finally` blocks.
  • nit: `test/unit/miner-portfolio-queue.test.ts:123` covers malformed repo, identifier, and priority inputs, but it does not cover an invalid DB path passed to `initPortfolioQueueStore`, which is part of the exported contract.
  • At `packages/gittensory-miner/lib/portfolio-queue.js:78`, consider statting/chmodding the parent directory or documenting that directory permissions are caller-owned when an existing config directory is supplied.
  • At `packages/gittensory-miner/lib/portfolio-queue.js:132`, track a `closed` boolean around `db.close()` so repeated cleanup calls become no-ops.
Signal Result Evidence
Code review ✅ No blockers 1 reviewer
Linked issue ✅ Linked #2292
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ❌ 5/25 Preflight is holding this PR: the review lane is unavailable, so it is not ready for automated review.
Contributor workload ✅ 10/10 Author activity: 27 registered-repo PR(s), 12 merged, 1 issue(s).
Contributor context ✅ Confirmed Gittensor contributor dhgoal; Gittensor profile; 27 PR(s), 1 issue(s).
Gate result ✅ Passing No configured blocker found.
Review context
  • Author: dhgoal
  • Role context: outside_contributor
  • Public audience mode: oss maintainer
  • Lane context: Repository registration is not available in the local Gittensory cache.
  • Public profile languages: not available
  • Official Gittensor activity: 27 PR(s), 1 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Await review-lane availability.
  • Refresh registry data or choose a registered active repo.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@dhgoal
dhgoal force-pushed the feat/miner-portfolio-queue-store branch from 50a0ada to 47d8593 Compare July 3, 2026 18:54
@dhgoal

dhgoal commented Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

Fixed the dequeue race. dequeueNext is now a single atomic statement — UPDATE … SET status='in_progress' WHERE rowid = (SELECT rowid … WHERE status='queued' ORDER BY priority DESC, enqueued_at ASC, rowid ASC LIMIT 1) RETURNING * — so two processes sharing the file can't both claim the same row (replacing the prior SELECT-then-UPDATE). Also added PRAGMA busy_timeout, an explicit string check in normalizeDbPath (proper invalid_portfolio_queue_db_path instead of a TypeError), a :memory: guard around mkdir/chmod, and listQueue/markDone invalid-input test coverage.

Add packages/gittensory-miner/lib/portfolio-queue.js: a 100% client-side
prioritized backlog of candidate work items across every repo the miner is
pointed at, backed by a local SQLite table, mirroring the run-state store
(lib/run-state.js). enqueue (upsert), dequeueNext (highest-priority first,
claims as in_progress, null on empty), listQueue (all or per-repo), and
markDone. Ordering is priority DESC, enqueued_at ASC, rowid ASC so the rowid
guarantees the insertion-order tie-break even on identical timestamps.

Persistence and read/write API only; priority is a placeholder numeric input
that later phases wire to the extracted reward-risk/scoring modules. The DB is
owner-only (0o600) and never leaves the machine.

Closes JSONbored#2292.

@loopover-orb loopover-orb Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gittensory approves — the gate is satisfied and CI is green.

@loopover-orb
loopover-orb Bot merged commit e54aa9a into JSONbored:main Jul 3, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contributor:flagged Contributor flagged for review by trust analysis. gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(miner-foundation): local portfolio/queue store for gittensory-miner

1 participant