Skip to content

fix(observability): fix Grafana dashboards showing no data after deploy - #5714

Merged
JSONbored merged 1 commit into
mainfrom
fix/grafana-dashboard-provisioning-disable-deletion
Jul 14, 2026
Merged

fix(observability): fix Grafana dashboards showing no data after deploy#5714
JSONbored merged 1 commit into
mainfrom
fix/grafana-dashboard-provisioning-disable-deletion

Conversation

@JSONbored

@JSONbored JSONbored commented Jul 14, 2026

Copy link
Copy Markdown
Owner

Summary

Two distinct, compounding bugs, both diagnosed and fixed live on the dedicated server before being turned into permanent code fixes:

1. disableDeletion: true orphaned every renamed dashboard forever

  • grafana/provisioning/dashboards/provider.yml had disableDeletion: true — Grafana's file-provisioner therefore never removes a dashboard from its own database even after its uid disappears from the provisioning files.
  • Every dashboard uid rename (the gittensory-* -> loopover-* rebrand already shipped in grafana/dashboards/*.json) permanently orphaned the old-uid entry. Grafana 13's unified-storage backend then hard-fails re-provisioning the new uid with a deprecatedInternalID already in use collision against that orphan — confirmed via Grafana's logs and its own sqlite resource table, which showed 9 dashboards still registered under pre-rebrand gittensory-* names while the files on disk had moved to loopover-*.
  • This recurs on every container restart/deploy until manually reconciled — no self-healing without this fix.
  • Fix: disableDeletion: false, matching how every other config-as-code source of truth in this stack already behaves.
  • A separate, pre-existing crash was also hit and fixed live (not part of this PR's diff, since it required a one-time production DB reconciliation, not a code change): Grafana's own data_source table still had the LoopoverDB SQLite datasource registered under uid gittensory-db while grafana/provisioning/datasources/sqlite.yml now declares loopover-db for the same datasource — this made Grafana's datasource-provisioning module fail outright on boot ("Datasource provisioning error: data source not found"), crash-looping the whole container. Fixed by updating that one row's uid to match; this can't recur on a fresh install since there'd be no legacy-uid row to collide with.

2. $__all used as a hand-rolled SQL "no filter" sentinel gets swallowed by Grafana's own macro-prefix convention

  • ai-usage.json/maintainer-reviews.json/miner-usage.json used allValue: "$__all" and compared it against a literal '$__all' to detect an unfiltered "All" selection: (${var:sqlstring} = '$__all' OR col = ${var:sqlstring}).
  • Confirmed live against the real Grafana + frser-sqlite-datasource instance: ${var:sqlstring} does not SQL-quote a value that itself starts with $__ (Grafana treats it as a macro reference, not literal data), so the substituted query carried the raw unquoted token $__all on both sides. SQLite then parsed that token as its own $__all named bind parameter, which was never supplied — every "All"-filtered panel either errored (missing named argument "__all") or silently returned zero rows, even with fresh underlying data (verified: the redacted reporting export had 61,741 rows with a last-event timestamp minutes old).
  • Fix: switch the sentinel to a plain string (__ALL__) that sqlstring quotes normally — verified with a direct sqlite3 execution of both the old and new pattern against the real reporting snapshot (old: 0 rows / error; new: full expected count).
  • Both dashboards' own regression tests had baked in the same wrong assumption about Grafana's substitution behavior in their simulation helpers, which is exactly why this shipped undetected — updated those too.

Scope

  • Config-as-code + dashboard JSON + regression tests only
  • No secrets/wallets/hotkeys/trust-scores/reward values
  • No site/, CNAME, **/lovable/**
  • No CHANGELOG.md edit

Validation

  • npx vitest run test/unit/selfhost-grafana-provisioning-config.test.ts test/unit/selfhost-grafana-ai-usage-dashboard.test.ts test/unit/selfhost-grafana-miner-usage-dashboard.test.ts test/unit/ai-usage-index.test.ts — all pass, including real sqlite3-execution assertions against the fixed query text
  • Live-verified end-to-end on the dedicated server: applied both fixes, restarted Grafana, confirmed via docker exec + Grafana's own sqlite tables that all 9 orphaned gittensory-* dashboards were cleaned up, every loopover-* dashboard (plus the newer loopover-miner-usage) provisioned with zero errors, and the AI-usage dashboard's SQL panels return correct row counts against the live reporting snapshot
  • git diff --check — clean

Safety

  • No secrets in code/comments/tests
  • Both fixes already verified safe and working in production before opening this PR

…ed dashboards

disableDeletion: true silently orphaned the old uid every time a dashboard's
uid changed in grafana/dashboards/*.json (e.g. the gittensory-* -> loopover-*
rebrand). Grafana's unified-storage backend then hard-fails re-provisioning
the new uid with an internal-id collision against the orphan, which recurred
on every subsequent deploy/restart until manually reconciled by hand -- the
actual cause of "most/all dashboards show no data" after an update. false
lets the provisioner clean up an orphaned dashboard the moment its file/uid
disappears, matching every other config-as-code source of truth in this
stack.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.05%. Comparing base (f7efb05) to head (034d611).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5714      +/-   ##
==========================================
+ Coverage   88.67%   95.05%   +6.37%     
==========================================
  Files         581      581              
  Lines       46164    46164              
  Branches    14810    14810              
==========================================
+ Hits        40936    43880    +2944     
+ Misses       3681     1523    -2158     
+ Partials     1547      761     -786     
Flag Coverage Δ
shard-1 43.55% <ø> (-0.43%) ⬇️
shard-2 35.84% <ø> (+0.13%) ⬆️
shard-3 32.22% <ø> (?)
shard-4 32.90% <ø> (-0.12%) ⬇️
shard-5 31.34% <ø> (-0.26%) ⬇️
shard-6 44.67% <ø> (+0.20%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 102 files with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@JSONbored JSONbored self-assigned this Jul 14, 2026
@JSONbored
JSONbored merged commit 7fa506c into main Jul 14, 2026
16 checks passed
@JSONbored
JSONbored deleted the fix/grafana-dashboard-provisioning-disable-deletion branch July 14, 2026 06:55
@JSONbored JSONbored changed the title fix(observability): let Grafana's dashboard provisioner delete orphaned dashboards fix(observability): fix Grafana dashboards showing no data after deploy Jul 14, 2026
@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant