feat(mnCount): move daily masternode-count history from external CSV into SQLite#16
feat(mnCount): move daily masternode-count history from external CSV into SQLite#16
Conversation
…into SQLite
Replaces the unsupervised standalone mnCount.js daemon (hardcoded RPC
creds, wrote /root/sysnode/data.csv out-of-repo) and the file-backed
routes/csvParser.js reader with first-party storage in
masternode_count_daily (date PK, INSERT OR IGNORE). The previous
arrangement broke /mnCount on any host where the CSV hadn't been
provisioned by a separate crontab outside the repo's control
(staging, fresh deploys, self-hosted mirrors).
- services/mnCountLogger.js: background writer. start() catches up
immediately if today's UTC row is missing, then arms for 00:00:05
UTC daily. Exponential backoff on RPC failure, clamped to never
overshoot the next midnight boundary. Idempotent start(), clean
stop(). Uses the existing cookie-auth rpcClient (no creds in code).
- lib/mnCountSeed.js: one-time idempotent seed from the committed
historical series at db/seeds/masternode-count.csv. Runs only when
the table is empty; wrapped in a single transaction so a mid-stream
parser failure rolls the whole seed back rather than poisoning
isEmpty() and blocking retry.
- lib/masternodeCountRepo.js: thin repo over the table, mirroring
the pattern already established by lib/reminderLog.js / lib/vaults.js.
- routes/mnCount.js: rewrote the old csvParser.js as a factory that
reads from the repo. Preserves the FE-visible [{date,users}] shape
so TrendChart.js (sysnode-info) keeps working byte-identically.
- db/migrations/001_init.sql: table added in-place per the pre-v1
rule in that file's header.
- db/seeds/masternode-count.csv: 2018-2026 historical series moved
from an out-of-repo location into the repo; loaded on first boot,
never re-read after that.
- mnCount.js (retired standalone daemon) removed from the working
tree; it was never tracked in git.
Tests: 33 new (10 seed, 19 logger, 4 route). Full suite 905/905.
Made-with: Cursor
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aca2f7e41e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| .finally(() => { | ||
| if (stopped) return; | ||
| schedule(msUntilNextMidnightUtc(now())); | ||
| }); |
There was a problem hiding this comment.
When start() runs, catchUpIfNeeded() failures are swallowed and the logger always schedules the next run at the next midnight boundary. If the process boots during the day and RPC is temporarily unavailable, there is no same-day retry; a later recovery still won’t be sampled until after midnight, so that UTC day is permanently missing from masternode_count_daily. This undermines the stated catch-up behavior for restart windows with transient startup failures.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Addressed in b79c18e.
Funneled the boot path through runAndReschedule() so all three outcomes are handled uniformly by the scheduler:
- today already recorded → arm for next midnight
- sample succeeds now → write, arm for next midnight
- sample fails now → backoff retry, clamped to stay inside this UTC day
The last case is what was missing: a transient RPC blip at boot (syscoind warming up after a joint pm2 restart, cookie rotation still propagating, etc.) now schedules a retry within the same day rather than deferring for 12-24h and losing today's row.
runAndReschedule() also picked up a fast-path skip when today's row is already there, so a spurious re-fire on the same UTC day cannot shadow the 00:00 snapshot with an afternoon value at the log layer. The repo's INSERT OR IGNORE would collapse the write anyway, but this avoids the needless RPC call and keeps the logs clean.
Tests added:
start() with RPC down retries inside the same UTC day (Codex PR16 P2)— asserts the scheduled delay is strictly less than time-to-next-midnight, and that after the retry succeeds the logger arms for next midnight rather than another retry.start() keeps backing off on repeated RPC failure, still inside today— three sequential failures all schedule retries inside today's UTC window.
…x P2)
start() previously routed through catchUpIfNeeded() and then
unconditionally armed for next midnight, so a transient RPC blip at
boot (syscoind warming up after a joint pm2 restart, cookie rotation
still propagating, etc.) would lose the current UTC day entirely —
the next sample wouldn't happen for 12-24h and today's row would
never be written.
Funnel the boot path through runAndReschedule() instead so all three
outcomes are handled uniformly:
- today already recorded → arm for next midnight
- sample succeeds now → write, arm for next midnight
- sample fails now → backoff retry, clamped to stay inside
this UTC day
runAndReschedule also gains a fast-path: if today's row is already
there it skips the RPC call and arms for next midnight, so a
spurious re-fire on the same UTC day cannot shadow the 00:00
snapshot with an afternoon value at the logging layer (the repo's
INSERT OR IGNORE would collapse the write anyway, but this is
cheaper and clearer).
catchUpIfNeeded() is preserved as a public method for one-shot
callers / existing tests that just want the "sample today if
missing" decision without touching the scheduler loop.
Tests: 2 new (boot-time recovery; repeated-failure backoff staying
inside today), 1 existing test tightened to assert both the skip
behavior AND the repo-level PK guarantee. Full suite 907/907.
Made-with: Cursor
|
@codex review. Don't worry about migrations |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b79c18edb1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| timer = null; | ||
| // Fire-and-forget: runAndReschedule owns its own rescheduling | ||
| // and error handling. | ||
| runAndReschedule(); |
There was a problem hiding this comment.
The timer callback invokes runAndReschedule() without awaiting or catching it, so any rejection escapes as an unhandled promise rejection and the logger stops rescheduling future writes. This is reachable when repo.getLatestDate() throws (it is outside the function's try/catch), for example on transient SQLite read failures, which can silently halt daily /mnCount updates until process restart.
Useful? React with 👍 / 👎.
| if (log) log('warn', 'mncount_seed_skip', { line: i + 1, reason: 'values' }); | ||
| continue; | ||
| } | ||
| yield { date: utcDateString(ts), total, recordedAt: ts }; |
There was a problem hiding this comment.
parseSeedCsv only checks that ts is finite and positive, then immediately calls utcDateString(ts). Extremely large numeric timestamps (still finite) make toISOString() throw RangeError, which aborts the seed transaction instead of skipping the bad row as intended by the parser's resilience rules. A single malformed timestamp in the seed file can therefore block all history from loading.
Useful? React with 👍 / 👎.
1 participant
Summary
Retires the unsupervised standalone
mnCount.jsdaemon (hardcoded RPC creds, wrote/root/sysnode/data.csvout-of-repo) and the file-backedroutes/csvParser.jsreader. Historical masternode-count data now lives in SQLite (masternode_count_daily, date PK +INSERT OR IGNORE) and is written by a first-class background service that's part of the normal pm2-managed backend lifecycle.This fixes the "Unable to load the latest network metrics right now" error on the staging homepage, which was the
/mnCountroute 500ing onEACCEStrying to read a CSV that doesn't exist on the staging box.What changed
services/mnCountLogger.js— daily writer.start()catches up immediately if today's UTC row is missing, then arms for00:00:05 UTCdaily (5s skew buffer). Exponential backoff on RPC failure, clamped to never overshoot the next midnight boundary. Idempotentstart(), cleanstop(). Uses the existing cookie-authrpcClient— zero credentials in code.lib/mnCountSeed.js— one-time idempotent seed from the committed historical series atdb/seeds/masternode-count.csv. Runs only when the table is empty; wrapped in a single transaction so a mid-stream parser failure rolls the whole seed back (otherwise a partial seed would poisonisEmpty()and block retry on the next boot).lib/masternodeCountRepo.js— thin repo over the table, mirroring the pattern already established bylib/reminderLog.js/lib/vaults.js.routes/mnCount.js— rewrote the oldroutes/csvParser.jsas a factory that reads from the repo. Preserves the FE-visible[{date,users}]shape soTrendChart.jsin sysnode-info keeps working byte-identically.db/migrations/001_init.sql— table added in-place per the pre-v1 rule in that file's header.db/seeds/masternode-count.csv— 2018-2026 historical series (2884 rows) moved from an out-of-repo location into the repo; loaded on first boot, never re-read after that. After deploy, the prod-side cron that was writing the legacy CSV can be retired (see test plan).mnCount.js(retired standalone daemon) removed from the working tree.Design notes
masternode_count= current only). A linear-interpolation or last-value-fill would be a lie; a gap shows up on the chart as a flat segment between the two bracketing points, which is the truthful representation.new Date()still report yesterday's UTC date and we'd stamp the wrong row.Test plan
Local:
Post-deploy on staging:
Follow-up ops (not in this PR):
Made with Cursor