improvement(knowledge): batch trigger dispatch, prune redundant DB roundtrips

[waleedlatif1]

Summary

• processDocumentsWithQueue now uses tasks.batchTrigger when Trigger.dev is available — collapses N HTTP roundtrips to ceil(N / 1000) per dispatch (a 100-doc connector sync drops from 100 calls to 1).
• Adds idempotency keys (doc-process-${documentId}-${requestId}) so retried dispatches don't double-enqueue.
• Per-batch isConnectorDeleted + isKnowledgeBaseDeleted checks merged into one checkSyncLiveness JOIN — one SELECT instead of two per sync batch.
• Dropped pre-upload isKnowledgeBaseDeleted checks from addDocument / updateDocument: the batch-boundary liveness check already catches pre-batch deletions, and the in-tx SELECT … FOR UPDATE is the authoritative point for mid-batch races.
• Deleted dead processDocumentsWithTrigger helper.

Type of Change

• Performance improvement

Testing

• All 208 KB unit tests pass (lib/knowledge/**, app/api/knowledge/**).
• No behavioral change in direct (non-Trigger.dev) mode — that path still uses per-doc parallel dispatch via dispatchDocumentProcessingJob.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		May 20, 2026 8:31pm

[cursor]

PR Summary

Medium Risk
Moderate risk because it changes background job dispatch semantics (batching, idempotency, partial-failure handling) and alters sync-time deletion checks, which could affect processing reliability under failure or deletion races.

Overview
Reduces connector sync DB roundtrips and improves processing dispatch throughput. The per-batch isConnectorDeleted + isKnowledgeBaseDeleted checks are replaced with a single checkSyncLiveness JOIN query, and sync now uses that combined result at batch boundaries and before stuck-doc retries.

Batches Trigger.dev processing enqueues. processDocumentsWithQueue now uses tasks.batchTrigger (chunked to 1,000) with per-document idempotency keys and logs partial failures, falling back to in-process execution when Trigger.dev isn’t available; the older per-doc dispatch helper and the unused processDocumentsWithTrigger are removed.

Prunes redundant pre-upload KB deletion checks. addDocument/updateDocument no longer do a separate pre-check for deleted knowledge bases, relying on the transactional FOR UPDATE liveness check instead.

^{Reviewed by Cursor Bugbot for commit a2cf183. Configure here.}

[greptile-apps]

Greptile Summary

This PR tightens the document-processing dispatch pipeline in two ways: it collapses per-document Trigger.dev tasks.trigger calls into chunked tasks.batchTrigger calls (cutting N HTTP roundtrips to ceil(N/1000)), and it merges the two per-batch DB liveness probes into a single JOIN query.

• dispatchViaBatchTrigger replaces the old per-doc tasks.trigger fan-out with tasks.batchTrigger, adds idempotency keys scoped to (documentId, requestId), and collects batchIds for dashboard traceability.
• checkSyncLiveness unifies the isConnectorDeleted + isKnowledgeBaseDeleted round-trips into one INNER JOIN query; both per-loop and post-loop call sites are updated to use the combined result.
• Pre-upload isKnowledgeBaseDeleted guards in addDocument/updateDocument are dropped; the transactional SELECT … FOR UPDATE in isKnowledgeBaseActiveInTx remains and is the authoritative mid-batch guard.

Confidence Score: 5/5

Safe to merge — the dispatch and liveness-check refactors are logically equivalent to the paths they replace, and the transactional FOR UPDATE guard that matters for correctness is untouched.

Both changed code paths (batch dispatch and liveness JOIN) preserve the observable contract: the same exceptions are thrown on deletion, the same documents land in Trigger.dev, and the direct-mode fallback calls the identical processDocumentAsync signature. The idempotency keys and typed batchTrigger generics are additive improvements with no downside risk.

No files require special attention.

Important Files Changed

Filename	Overview
apps/sim/lib/knowledge/documents/service.ts	Replaces per-doc trigger fan-out with chunked batchTrigger; adds typed generics, idempotency keys, and batchId logging. Direct (non-Trigger) path refactored into dispatchInProcess with identical semantics to the old code.
apps/sim/lib/knowledge/connectors/sync-engine.ts	Merges isConnectorDeleted + isKnowledgeBaseDeleted into a single JOIN query (checkSyncLiveness). Removes redundant pre-upload KB checks; transactional FOR UPDATE guard in addDocument/updateDocument is retained.

_{Reviews (2): Last reviewed commit: "improvement(knowledge): thread requestId..." | Re-trigger Greptile}

[waleedlatif1]

@greptile

[waleedlatif1]

@cursor review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit a2cf183. Configure here.}