fix(observability): fix Grafana dashboards showing no data after deploy - #5714
Merged
JSONbored merged 1 commit intoJul 14, 2026
Merged
Conversation
…ed dashboards disableDeletion: true silently orphaned the old uid every time a dashboard's uid changed in grafana/dashboards/*.json (e.g. the gittensory-* -> loopover-* rebrand). Grafana's unified-storage backend then hard-fails re-provisioning the new uid with an internal-id collision against the orphan, which recurred on every subsequent deploy/restart until manually reconciled by hand -- the actual cause of "most/all dashboards show no data" after an update. false lets the provisioner clean up an orphaned dashboard the moment its file/uid disappears, matching every other config-as-code source of truth in this stack.
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5714 +/- ##
==========================================
+ Coverage 88.67% 95.05% +6.37%
==========================================
Files 581 581
Lines 46164 46164
Branches 14810 14810
==========================================
+ Hits 40936 43880 +2944
+ Misses 3681 1523 -2158
+ Partials 1547 761 -786
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
JSONbored
deleted the
fix/grafana-dashboard-provisioning-disable-deletion
branch
July 14, 2026 06:55
6 tasks
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two distinct, compounding bugs, both diagnosed and fixed live on the dedicated server before being turned into permanent code fixes:
1.
disableDeletion: trueorphaned every renamed dashboard forevergrafana/provisioning/dashboards/provider.ymlhaddisableDeletion: true— Grafana's file-provisioner therefore never removes a dashboard from its own database even after itsuiddisappears from the provisioning files.uidrename (thegittensory-*->loopover-*rebrand already shipped ingrafana/dashboards/*.json) permanently orphaned the old-uid entry. Grafana 13's unified-storage backend then hard-fails re-provisioning the new uid with adeprecatedInternalID already in usecollision against that orphan — confirmed via Grafana's logs and its own sqliteresourcetable, which showed 9 dashboards still registered under pre-rebrandgittensory-*names while the files on disk had moved toloopover-*.disableDeletion: false, matching how every other config-as-code source of truth in this stack already behaves.data_sourcetable still had theLoopoverDBSQLite datasource registered under uidgittensory-dbwhilegrafana/provisioning/datasources/sqlite.ymlnow declaresloopover-dbfor the same datasource — this made Grafana's datasource-provisioning module fail outright on boot ("Datasource provisioning error: data source not found"), crash-looping the whole container. Fixed by updating that one row'suidto match; this can't recur on a fresh install since there'd be no legacy-uid row to collide with.2.
$__allused as a hand-rolled SQL "no filter" sentinel gets swallowed by Grafana's own macro-prefix conventionai-usage.json/maintainer-reviews.json/miner-usage.jsonusedallValue: "$__all"and compared it against a literal'$__all'to detect an unfiltered "All" selection:(${var:sqlstring} = '$__all' OR col = ${var:sqlstring}).frser-sqlite-datasourceinstance:${var:sqlstring}does not SQL-quote a value that itself starts with$__(Grafana treats it as a macro reference, not literal data), so the substituted query carried the raw unquoted token$__allon both sides. SQLite then parsed that token as its own$__allnamed bind parameter, which was never supplied — every "All"-filtered panel either errored (missing named argument "__all") or silently returned zero rows, even with fresh underlying data (verified: the redacted reporting export had 61,741 rows with a last-event timestamp minutes old).__ALL__) thatsqlstringquotes normally — verified with a directsqlite3execution of both the old and new pattern against the real reporting snapshot (old: 0 rows / error; new: full expected count).Scope
site/,CNAME,**/lovable/**CHANGELOG.mdeditValidation
npx vitest run test/unit/selfhost-grafana-provisioning-config.test.ts test/unit/selfhost-grafana-ai-usage-dashboard.test.ts test/unit/selfhost-grafana-miner-usage-dashboard.test.ts test/unit/ai-usage-index.test.ts— all pass, including realsqlite3-execution assertions against the fixed query textdocker exec+ Grafana's own sqlite tables that all 9 orphanedgittensory-*dashboards were cleaned up, everyloopover-*dashboard (plus the newerloopover-miner-usage) provisioned with zero errors, and the AI-usage dashboard's SQL panels return correct row counts against the live reporting snapshotgit diff --check— cleanSafety