fix(search): preserve indexed vector retrieval and literal source text

[icecrasher321]

Summary

• Keep broad vector retrieval driven by the nearest-neighbor index when permission estimates are stale. Preserve authorization before the candidate limit, existing ANN settings, full-vector reranking, and a compact exact fallback for underfilled scans.
• Preserve literal source text in canonical stored .txt artifacts so HTML application shells and RTF snippets can be indexed without being interpreted as rendered documents.
• Extend the existing retrieval harness to cover the 512-dimensional candidate path, interleaved tenants, stale permission estimates, concurrent organization searches, and selective-scope recall.

Type of Change

• Bug fix

Testing

• The 206,000-chunk synthetic integration benchmark passed 26 tests on 2 CPUs / 1 GiB: 40 normal searches with no retrieval timeouts, 100% recall in all 11 exact-reference comparisons, and a maximum vector stage of 669 ms. Both filled ANN scans and selective exact fallbacks were exercised.
• Organization searches after shared-buffer eviction took 3.45–3.99 s overall, dominated by keyword retrieval. The operating-system cache was not cleared; provider HTTP responses were controlled while PostgreSQL, Redis, authorization, and application search ran normally.
• Both concurrency scenarios passed again with distinct query vectors. Independent SQL checks verified unused fallback CTEs on filled ANN scans and a single compact scoring scan with a spilling join at 64 KiB work_mem.
• Search query, budget, and diagnostics tests: 68 passed.
• Parser, document ingestion, and connector regression suites: 1,045 passed.
• Focused tests after merging latest staging: 254 passed.
• App type checking, repository lint, all 46 audits, block registry audit, and generated-artifact checks passed.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Sep 17, 2026 12:02am UTC

[icecrasher321]

@greptile

[icecrasher321]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

[greptile-apps]

Confidence Score: 5/5

The reviewed changes appear safe to merge, with no new actionable correctness, security, migration, or repository-rule issue established.

Summary

This PR preserves literal source markup in canonical text artifacts and restructures authorized vector retrieval so ANN traversal remains index-driven before exact reranking. Changes since the previous review also strengthen connector permission-failure persistence, sync health reporting, pagination validation, and member-sync diagnostics.

• Routes canonical .txt artifacts through literal parsing while retaining normal HTML/RTF handling elsewhere.
• Uses the 512-dimensional ANN projection for broad authorized candidate retrieval and full vectors for reranking.
• Expands the integration benchmark to exercise stale estimates, selective scopes, concurrent organization searches, and production-equivalent HNSW settings.
• Persists incomplete permission verification across resumable connector listings without advancing successful-sync freshness.
• Adds member-sync failure counters, continuation-aware status reporting, and organization-overview polling across worker handoffs.
• Rejects malformed or cyclic Confluence v2 permission pagination rather than publishing partial grants.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Q[Search query] --> E[Query embedding]
  E --> P[512-dimensional ANN projection]
  P --> A[Per-candidate document authorization]
  A --> L{Candidate limit filled?}
  L -->|Yes| C[Compact candidate IDs]
  L -->|No| F[Exact filtered projection fallback]
  F --> C
  C --> R[Full-vector reranking]
  R --> H[Live authorization and hydration]
  H --> O[Search results]

Loading

_{Reviews (4) · Last reviewed commit: "fix(search): retain ANN recall settings ..."}

[icecrasher321]

@greptile

[icecrasher321]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

All reported issues were addressed across 9 files

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[icecrasher321]

@greptile

[icecrasher321]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

No issues found across 9 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

_{Re-trigger cubic}

[icecrasher321]

@greptile

[icecrasher321]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

No issues found across 9 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

_{Re-trigger cubic}