Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -523,6 +523,16 @@ REDIS_URL=redis://redis:6379 # REQUIRED for the self-host review
# data source after Grafana is up with: ./scripts/setup-github-datasource.sh (reads GITHUB_TOKEN below).
# GITHUB_TOKEN=<read-only-github-token> # fine-grained PAT: Pull requests:read, Issues:read, Contents:read
#
# In-Grafana Sentry error/issue visualization (#5369, distinct from SENTRY_DSN above, which is for error
# REPORTING). Install is automatic (GF_INSTALL_PLUGINS); add the data source after Grafana is up with:
# ./scripts/setup-sentry-datasource.sh (reads the two vars below). SENTRY_DSN is NOT reusable here -- a DSN
# authenticates event ingestion, not the read/query API this datasource needs.
# SENTRY_API_TOKEN=<sentry-internal-integration-token> # Sentry -> Settings -> Developer Settings -> Custom
# # Integrations -> New Internal Integration, with Read access on
# # Project, Issue & Event, and Organization scopes
# SENTRY_ORG_SLUG=<your-sentry-org-slug>
# SENTRY_API_URL=https://sentry.io # override only for a self-hosted Sentry instance
#
# Discord notifications. Alertmanager → Discord (system/stack alerts) is configured in alertmanager/alertmanager.yml.
# The ENGINE posts a per-repo review summary when it publishes a review — set a per-repo map and/or a global fallback:
# DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/... # global fallback for any repo without its own
Expand Down
44 changes: 44 additions & 0 deletions apps/gittensory-ui/src/routes/docs.self-hosting-operations.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -790,6 +790,50 @@ SENTRY_RELEASE=gittensory-selfhost@2026.07.05
same subsystem.
</p>

<h2>Grafana Sentry data source (in-Grafana issue visualization)</h2>
<p>
Query recent Sentry issues, top issues by event count, and error-volume trend directly in
Grafana — no more switching tabs to check Sentry, and errors line up in time with the rest
of the stack&apos;s metrics/logs/traces. This is read-only visualization; alert routing to
Sentry/Discord/Slack is a separate, unrelated concern covered above.
</p>
<Callout variant="warn" title="SENTRY_DSN is NOT reusable here">
The <code>SENTRY_DSN</code> above authenticates event <strong>ingestion</strong> (sending
errors to Sentry), not the read/query API this data source needs. You need a separate{" "}
<strong>Sentry Internal Integration token</strong>: Sentry → Settings → Developer Settings →
Custom Integrations → New Internal Integration (requires an Admin/Manager/Owner role in
Sentry), with <strong>Read</strong> access on the <strong>Project</strong>,{" "}
<strong>Issue &amp; Event</strong>, and <strong>Organization</strong> resource scopes.
</Callout>
<p>
The{" "}
<a
href="https://grafana.com/grafana/plugins/grafana-sentry-datasource/"
target="_blank"
rel="noreferrer"
>
grafana-sentry-datasource
</a>{" "}
plugin installs automatically (<code>GF_INSTALL_PLUGINS</code>, same mechanism as the GitHub
data source below). Add the data source itself after Grafana is up — a backend datasource
whose token isn&apos;t ready at Grafana&apos;s own boot time would crash file-based
provisioning, so this one is added over the API instead, exactly like the GitHub data
source:
</p>
<CodeBlock
filename=".env"
code={`SENTRY_API_TOKEN=<your-sentry-internal-integration-token>
SENTRY_ORG_SLUG=<your-sentry-org-slug>
# SENTRY_API_URL=https://sentry.io # override only for a self-hosted Sentry instance`}
/>
<CodeBlock lang="bash" code={`./scripts/setup-sentry-datasource.sh`} />
<p>
The script is idempotent — safe to re-run after rotating the token. Open the{" "}
<strong>Sentry issues</strong> dashboard once it succeeds. Same trade-off as the GitHub data
source: this one is API-managed, so it stays editable via the Grafana UI rather than locked
read-only like the file-provisioned data sources.
</p>

<h2>Re-gate sweeps (agent-regate-sweep)</h2>
<p>
Live PR review is webhook-driven, but open PRs still need periodic re-evaluation — the base
Expand Down
4 changes: 3 additions & 1 deletion docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -762,7 +762,9 @@ services:
# check lives in entrypoint below. Runtime still fails closed unless the operator sets GRAFANA_ADMIN_PASSWORD.
GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:-${GRAFANA_LOCAL_SMOKE_PASSWORD:-}}
GF_USERS_ALLOW_SIGN_UP: "false"
GF_INSTALL_PLUGINS: frser-sqlite-datasource,grafana-github-datasource
# grafana-sentry-datasource (#5369): first-party Grafana Labs plugin, confirmed Grafana-signed and
# installs cleanly on this pinned Grafana version (verified live, not assumed).
GF_INSTALL_PLUGINS: frser-sqlite-datasource,grafana-github-datasource,grafana-sentry-datasource
# Read-only fine-grained PAT for the GitHub data source provisioning ($GITHUB_TOKEN expansion). From .env.
GITHUB_TOKEN: "${GITHUB_TOKEN:-}"
deploy:
Expand Down
2 changes: 1 addition & 1 deletion grafana/dashboards/resource-hub.json
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@
"gridPos": { "h": 13, "w": 12, "x": 12, "y": 0 },
"options": {
"mode": "markdown",
"content": "## 📊 Dashboards\n- **[Upstream PRs & issues (GitHub)](/d/gittensory-github)** — live, accurate census + open-PR triage (GitHub API).\n- **[Reviews & PRs (maintainer)](/d/gittensory-maintainer)** — gittensory's own review activity + reviewed-PR log.\n- **[AI usage](/d/gittensory-ai-usage)** — durable cross-provider ai_usage_events (filterable by provider/feature/model), live Prometheus counters, and Claude Code's own OTEL session telemetry, all in one place.\n- **[Gittensory (infra)](/d/gittensory)** — queue, jobs, HTTP, GitHub API cache/rate limits.\n- **[GPU metrics](/d/gittensory-gpu)** — utilization/VRAM for a self-hosted Ollama GPU box.\n- **[Infra health](/d/gittensory-infra-health)** — host CPU/mem/disk/network (node-exporter), per-container resource usage (cAdvisor), Redis, Qdrant, and whether the observability stack itself is up.\n- **[REES (review-enrichment)](/d/gittensory-rees)** — request outcomes/latency and per-analyzer run/timeout/degrade rates for the optional `--profile rees` service.\n- **[Browserless (visual review)](/d/gittensory-browserless)** — queue depth, concurrency, and rejection/error/timeout rate for the optional `--profile visual-review` screenshot service.\n\n## 📈 Metrics & logs\n- **Prometheus** — [targets](http://localhost:9090/targets) · [graph](http://localhost:9090)\n- **Alertmanager** — [alerts](http://localhost:9093)\n- **Loki** — query in [Explore](/explore) (pick the *Loki* datasource), e.g. `{compose_service=\"gittensory\"}`\n- **Sentry** — release/source-map enriched errors. Edit the dashboard link if your project URL differs.\n\n## 🩺 Quick health checks\n| What | Where |\n|---|---|\n| App serving | `GET /ready` → 200 |\n| AI wired | boot log `selfhost_ai_provider` |\n| Embeds wired | boot log `selfhost_embed_provider` |\n| Vectors wired | boot log `selfhost_vectorize` |\n| Token spend | **[AI usage](/d/gittensory-ai-usage)** dashboard |\n\n## 📚 Docs\n- [Maintainer self-hosting](https://gittensory.aethereal.dev/docs/maintainer-self-hosting) — setup, configuration, AI, REES, RAG, operations, and troubleshooting."
"content": "## 📊 Dashboards\n- **[Upstream PRs & issues (GitHub)](/d/gittensory-github)** — live, accurate census + open-PR triage (GitHub API).\n- **[Reviews & PRs (maintainer)](/d/gittensory-maintainer)** — gittensory's own review activity + reviewed-PR log.\n- **[AI usage](/d/gittensory-ai-usage)** — durable cross-provider ai_usage_events (filterable by provider/feature/model), live Prometheus counters, and Claude Code's own OTEL session telemetry, all in one place.\n- **[Gittensory (infra)](/d/gittensory)** — queue, jobs, HTTP, GitHub API cache/rate limits.\n- **[GPU metrics](/d/gittensory-gpu)** — utilization/VRAM for a self-hosted Ollama GPU box.\n- **[Infra health](/d/gittensory-infra-health)** — host CPU/mem/disk/network (node-exporter), per-container resource usage (cAdvisor), Redis, Qdrant, and whether the observability stack itself is up.\n- **[REES (review-enrichment)](/d/gittensory-rees)** — request outcomes/latency and per-analyzer run/timeout/degrade rates for the optional `--profile rees` service.\n- **[Browserless (visual review)](/d/gittensory-browserless)** — queue depth, concurrency, and rejection/error/timeout rate for the optional `--profile visual-review` screenshot service.\n- **[Sentry issues](/d/gittensory-sentry)** — recent unresolved issues, top issues by event count, and error-volume trend, queried live from Sentry (`scripts/setup-sentry-datasource.sh`). The plain link below still opens Sentry itself for actions this read-only view can't do (resolving/assigning issues).\n\n## 📈 Metrics & logs\n- **Prometheus** — [targets](http://localhost:9090/targets) · [graph](http://localhost:9090)\n- **Alertmanager** — [alerts](http://localhost:9093)\n- **Loki** — query in [Explore](/explore) (pick the *Loki* datasource), e.g. `{compose_service=\"gittensory\"}`\n- **Sentry** — release/source-map enriched errors. Edit the dashboard link if your project URL differs.\n\n## 🩺 Quick health checks\n| What | Where |\n|---|---|\n| App serving | `GET /ready` → 200 |\n| AI wired | boot log `selfhost_ai_provider` |\n| Embeds wired | boot log `selfhost_embed_provider` |\n| Vectors wired | boot log `selfhost_vectorize` |\n| Token spend | **[AI usage](/d/gittensory-ai-usage)** dashboard |\n\n## 📚 Docs\n- [Maintainer self-hosting](https://gittensory.aethereal.dev/docs/maintainer-self-hosting) — setup, configuration, AI, REES, RAG, operations, and troubleshooting."
}
}
]
Expand Down
99 changes: 99 additions & 0 deletions grafana/dashboards/sentry-issues.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
{
"__inputs": [],
"__requires": [
{ "type": "grafana", "id": "grafana", "name": "Grafana", "version": "10.0.0" },
{ "type": "datasource", "id": "grafana-sentry-datasource", "name": "Sentry", "version": "2.2.4" }
],
"annotations": { "list": [] },
"editable": false,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"id": null,
"links": [],
"description": "Sentry error/issue data queried directly in Grafana (#5369) — no more leaving Grafana to check Sentry, and errors can be correlated in time with the rest of the stack's metrics/logs/traces. Requires the Sentry data source configured via scripts/setup-sentry-datasource.sh (SENTRY_API_TOKEN + SENTRY_ORG_SLUG — a separate credential from SENTRY_DSN). Distinct from #5007 (Alertmanager/Sentry notification routing) — this is read-only visualization.",
"panels": [
{ "collapsed": false, "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 }, "id": 1, "title": "Issue volume", "type": "row" },
{
"datasource": { "type": "grafana-sentry-datasource", "uid": "${DS_SENTRY}" },
"fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "lineWidth": 2, "fillOpacity": 10 } } },
"gridPos": { "h": 9, "w": 24, "x": 0, "y": 1 },
"id": 2,
"options": { "legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom" }, "tooltip": { "mode": "multi", "sort": "desc" } },
"title": "Error events over time",
"description": "Event count trend for event.type:error, scoped to the dashboard's own time range.",
"type": "timeseries",
"targets": [
{
"refId": "A",
"datasource": { "type": "grafana-sentry-datasource", "uid": "${DS_SENTRY}" },
"queryType": "eventsStats",
"projectIds": [],
"environments": [],
"eventsStatsQuery": "event.type:error",
"eventsStatsYAxis": ["count()"],
"eventsStatsGroups": []
}
]
},

{ "collapsed": false, "gridPos": { "h": 1, "w": 24, "x": 0, "y": 10 }, "id": 10, "title": "Issues", "type": "row" },
{
"datasource": { "type": "grafana-sentry-datasource", "uid": "${DS_SENTRY}" },
"gridPos": { "h": 10, "w": 12, "x": 0, "y": 11 },
"id": 11,
"title": "Recent unresolved issues",
"description": "Most recently seen unresolved issues. Add a project filter directly to this panel's target (projectIds) once you know your Sentry numeric project id(s) -- left empty here queries across every project the token can read.",
"type": "table",
"options": { "showHeader": true, "cellHeight": "sm" },
"fieldConfig": { "defaults": { "custom": { "align": "auto", "filterable": true } } },
"targets": [
{
"refId": "A",
"datasource": { "type": "grafana-sentry-datasource", "uid": "${DS_SENTRY}" },
"queryType": "issues",
"projectIds": [],
"environments": [],
"issuesQuery": "is:unresolved",
"issuesSort": "date",
"issuesLimit": 20
}
]
},
{
"datasource": { "type": "grafana-sentry-datasource", "uid": "${DS_SENTRY}" },
"gridPos": { "h": 10, "w": 12, "x": 12, "y": 11 },
"id": 12,
"title": "Top issues by event count",
"description": "Unresolved issues sorted by total event volume -- the highest-impact issues to triage first, not just the newest.",
"type": "table",
"options": { "showHeader": true, "cellHeight": "sm" },
"fieldConfig": { "defaults": { "custom": { "align": "auto", "filterable": true } } },
"targets": [
{
"refId": "A",
"datasource": { "type": "grafana-sentry-datasource", "uid": "${DS_SENTRY}" },
"queryType": "issues",
"projectIds": [],
"environments": [],
"issuesQuery": "is:unresolved",
"issuesSort": "freq",
"issuesLimit": 10
}
]
}
],
"refresh": "5m",
"schemaVersion": 38,
"tags": ["gittensory", "sentry", "observability"],
"templating": {
"list": [
{ "current": {}, "hide": 0, "includeAll": false, "name": "DS_SENTRY", "options": [], "query": "grafana-sentry-datasource", "refresh": 1, "type": "datasource" }
]
},
"time": { "from": "now-24h", "to": "now" },
"timepicker": {},
"timezone": "browser",
"title": "Gittensory — Sentry issues",
"uid": "gittensory-sentry",
"version": 1
}
59 changes: 59 additions & 0 deletions scripts/setup-sentry-datasource.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
#!/usr/bin/env bash
# Add (or update) the Sentry data source in Grafana via the API — in-Grafana error/issue visualization,
# correlated in time with the rest of the stack (#5369). Distinct from #5007 (Alertmanager/Sentry
# notification routing): this is read-only visualization, not paging.
#
# Done over the API rather than file-provisioning, for the exact reason scripts/setup-github-datasource.sh
# already documents: a backend datasource whose plugin/token isn't ready at boot would crash Grafana's
# provisioning, so we add it after Grafana is up.
#
# IMPORTANT: SENTRY_DSN (already used for this stack's own error REPORTING to Sentry, see the self-hosting
# docs) is NOT sufficient here and cannot be reused — a DSN authenticates event *ingestion*, not the
# read/query API this datasource needs. You need a separate Sentry "Internal Integration" token (Sentry →
# Settings → Developer Settings → Custom Integrations → New Internal Integration, requires an Admin/Manager/
# Owner role) with Read access on the Project, Issue & Event, and Organization resource scopes.
#
# KNOWN, ACCEPTED trade-off: same as the GitHub datasource — API-managed, so NOT locked read-only via
# Grafana's `readOnly` field (that flag can only be set by file provisioning, not the datasource API).
#
# Prereqs: --profile observability running, the grafana-sentry-datasource plugin installed
# (GF_INSTALL_PLUGINS, confirmed Grafana-signed and Grafana-13.x-compatible), and a Sentry Internal
# Integration auth token with the scopes above.
#
# Usage:
# SENTRY_API_TOKEN=<sentry-internal-integration-token> SENTRY_ORG_SLUG=<your-org-slug> \
# GRAFANA_ADMIN_PASSWORD=... ./scripts/setup-sentry-datasource.sh
# # or rely on values already in ./.env (SENTRY_API_TOKEN, SENTRY_ORG_SLUG, GRAFANA_ADMIN_PASSWORD)
set -euo pipefail

GRAFANA_URL="${GRAFANA_URL:-http://localhost:3000}"
[ -f .env ] && { set -a; . ./.env; set +a; }
: "${SENTRY_API_TOKEN:?Set SENTRY_API_TOKEN (a Sentry Internal Integration token, NOT your SENTRY_DSN) in the environment or .env}"
: "${SENTRY_ORG_SLUG:?Set SENTRY_ORG_SLUG (your Sentry organization slug) in the environment or .env}"
: "${GRAFANA_ADMIN_PASSWORD:?Set GRAFANA_ADMIN_PASSWORD in the environment or .env}"
AUTH="admin:${GRAFANA_ADMIN_PASSWORD}"
# https://sentry.io for Sentry SaaS; override for a self-hosted Sentry instance.
SENTRY_API_URL="${SENTRY_API_URL:-https://sentry.io}"

payload() {
cat <<JSON
{ "name": "Sentry", "type": "grafana-sentry-datasource", "uid": "sentry", "access": "proxy",
"isDefault": false, "jsonData": { "url": "${SENTRY_API_URL}", "orgSlug": "${SENTRY_ORG_SLUG}" },
"secureJsonData": { "authToken": "${SENTRY_API_TOKEN}" } }
JSON
}

# Idempotent: update in place if a datasource with uid "sentry" already exists, else create it.
if curl -sf -u "$AUTH" "$GRAFANA_URL/api/datasources/uid/sentry" >/dev/null 2>&1; then
echo "Updating existing Sentry data source…"
curl -sf -u "$AUTH" -H 'content-type: application/json' -X PUT \
"$GRAFANA_URL/api/datasources/uid/sentry" -d "$(payload)" >/dev/null
else
echo "Creating Sentry data source…"
curl -sf -u "$AUTH" -H 'content-type: application/json' -X POST \
"$GRAFANA_URL/api/datasources" -d "$(payload)" >/dev/null
fi

echo "Done. Verifying health…"
curl -sf -u "$AUTH" -X POST "$GRAFANA_URL/api/datasources/uid/sentry/health" 2>/dev/null \
| grep -q '"status":"OK"' && echo "✓ Sentry data source healthy" || echo "⚠ Added, but health check did not return OK — verify SENTRY_API_TOKEN's scopes and SENTRY_ORG_SLUG."
9 changes: 9 additions & 0 deletions test/unit/docs-selfhost-sentry-observability.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -45,4 +45,13 @@ describe("self-host Sentry observability docs (#1824)", () => {
expect(operations).toContain("selfhost_backup_advisory");
expect(operations).toContain("scheduled-loop");
});

it("documents the in-Grafana Sentry data source (#5369) as a separate credential from SENTRY_DSN", () => {
expect(operations).toContain("Grafana Sentry data source");
expect(operations).toContain("SENTRY_API_TOKEN");
expect(operations).toContain("SENTRY_ORG_SLUG");
expect(operations).toContain("setup-sentry-datasource.sh");
expect(operations).toContain("Internal Integration");
expect(operations).toMatch(/SENTRY_DSN.*NOT reusable|not reusable.*SENTRY_DSN/i);
});
});
Loading
Loading