fix(review): drop whitespace-only RAG chunks and surface real embed errors

[JSONbored]

Summary

• Fixes fix(review): RAG embedding calls rejected with HTTP 400 (485 Sentry events, ongoing) #4996: ai_embed_http_400 fired 485 times over 2 weeks, still ongoing at filing time.
• Sentry Seer's summary for this issue referenced a 401/403 circuit-breaker — didn't match the actual error (400, Bad Request, not an auth code), so I ignored it and traced the real embed path instead. Verified live on the self-host box: AI_EMBED_MODEL=bge-m3:latest is correctly configured and the model is genuinely pulled in Ollama — ruling out a config/model-availability mismatch.
• Two fixes:
  1. chunkFile only validated that the WHOLE file was non-empty before splitting — an individual slice from newlineChunks/chunkJsTs can land entirely inside a run of blank/whitespace-only lines and reach the embed API as an effectively-empty string, the most plausible cause of a 400 from Ollama's OpenAI-compatible /embeddings endpoint. Filtered after chunking (single centralized guard covering both chunkers), preserving each surviving chunk's original chunkIndex/id rather than renumbering.
  2. The embed error path threw only the bare status code (ai_embed_http_400), discarding the response body entirely — diagnosing this required live production archaeology (SSH, cross-referencing box config) instead of just reading the error. Now captures a bounded (300 char) response-body detail, best-effort so a body-read failure never masks the original status.

Scope

• The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
• This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
• This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
• I linked a currently open issue this PR resolves (Closes #4996).

Validation

• git diff --check
• npm run actionlint
• npm run typecheck
• npm run test:coverage locally
• npm run test:workers
• npm run build:mcp
• npm run test:mcp-pack
• npm run ui:openapi:check
• npm run ui:lint
• npm run ui:typecheck
• npm run ui:build
• npm audit --audit-level=moderate
• New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries

If any required check was skipped, explain why:

• test:coverage (full unsharded): not run end-to-end — ran scoped vitest --coverage for test/unit/rag.test.ts + test/unit/selfhost-ai.test.ts (244 tests) and confirmed via lcov that every changed line/branch is covered.
• test:workers / build:mcp / test:mcp-pack / ui:lint / ui:typecheck / ui:build: not run — this change touches only src/review/rag.ts and src/selfhost/ai.ts; no Worker-pool, MCP, or UI surface changed.

Safety

• No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
• Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
• Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests. (N/A — no auth surface touched; the captured error-body detail is bounded/best-effort and never includes secrets, since it's this instance's own local embedding endpoint's response.)
• API/OpenAPI/MCP behavior is updated and tested where needed. (N/A.)
• UI changes use live API data or real empty/error/loading states. (N/A.)
• Visible UI changes include a UI Evidence section. (N/A.)
• Public docs/changelogs are updated where needed. (N/A — internal engine behavior; changelog is not edited in a normal PR.)

Notes

Part of a batch of 13 bug fixes filed from a Sentry-issue triage this session (#4994–#5006). This is #3 by impact.

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.36%. Comparing base (19c8acb) to head (881f138).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #5029   +/-   ##
=======================================
  Coverage   94.36%   94.36%           
=======================================
  Files         457      457           
  Lines       39236    39236           
  Branches    14313    14313           
=======================================
  Hits        37024    37024           
  Misses       1558     1558           
  Partials      654      654           

Flag	Coverage Δ
shard-1	46.48% <40.00%> (-0.29%)	⬇️
shard-2	33.66% <100.00%> (+0.26%)	⬆️
shard-3	31.37% <0.00%> (+0.32%)	⬆️
shard-4	32.59% <0.00%> (-0.59%)	⬇️
shard-5	34.47% <0.00%> (+1.00%)	⬆️
shard-6	45.33% <0.00%> (+0.20%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/review/rag.ts	100.00% <100.00%> (ø)
src/selfhost/ai.ts	98.04% <100.00%> (+<0.01%)	⬆️

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[loopover-orb]

Important

🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪

🔍 Gittensory is reviewing…

AI analysis is in progress. This comment will update when the review is complete.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed · 🟪 Reviewing}