Background relation resolution reads notes before async materialization

[phernandez]

Summary

On current main, an accepted write_note mutation can trigger relation resolution before its Markdown file has been materialized. The MCP call correctly returns 202/Created, but the debounced background relation-resolution pass may reindex an affected source entity while that entity is still in the valid accepted-but-file-pending state.

The reindex path supplies no preloaded content, falls back to reading Markdown from disk, and logs FileNotFoundError. Materialization and the watcher later recover, so the write eventually becomes searchable, but the expected pending window is currently treated as a background task failure.

Observed on main commit 60408ad7 while rerunning the release write-load harness from #1021 against published v0.22.1.

Execution path

1. knowledge_router.create_entity() persists the accepted NoteContent mutation.
2. LocalNoteContentMaterializationProvider.materialize_write_change() schedules Markdown materialization off the acceptance path and returns immediately.
3. _schedule_post_write_followups() schedules debounced relation resolution for the project.
4. RepositoryRelationResolutionRuntime.resolve_relations() resolves inbound references and reindexes affected source entities through entity_indexer.index_entities(...).
5. SearchService.index_entities() calls index_entity_data(entity) without content.
6. index_entity_markdown() sees content is None and calls file_service.read_entity_content(entity).
7. If materialization is still pending, the Markdown path does not exist and the background task raises FileNotFoundError.

Representative stack:

LocalRelationResolutionScheduler._resolve_after_debounce
  -> resolve_project_relations
  -> RepositoryRelationResolutionRuntime.resolve_relations
  -> SearchService.index_entities
  -> SearchService.index_entity_data(entity, content=None)
  -> SearchService.index_entity_markdown
  -> FileService.read_entity_content
  -> Path.read_text
  -> FileNotFoundError

Relevant code:

• src/basic_memory/api/v2/routers/knowledge_router.py::_schedule_post_write_followups
• src/basic_memory/index/note_content_materialization.py::LocalNoteContentMaterializationProvider.materialize_write_change
• src/basic_memory/indexing/relation_resolution.py::RepositoryRelationResolutionRuntime.resolve_relations
• src/basic_memory/services/search_service.py::SearchService.index_entities
• src/basic_memory/services/search_service.py::SearchService.index_entity_markdown

Benchmark evidence

The release benchmark used three fresh-runtime repeats per backend, 100 notes per concurrency level, concurrency 1/8/32/64, and eight warmups.

All 36 valid measurement rows had zero caller-visible write errors and every measured level eventually became searchable. However, every candidate run logged transient background search-index FileNotFoundErrors:

• SQLite: 27, 25, and 26 failures across the three repeats
• Postgres: 15, 15, and 10 failures across the three repeats
• Published v0.22.1 SQLite: zero

This is therefore reproducible under bursty write load. It adds error noise and redundant recovery work, and it makes genuine missing-file defects harder to distinguish from a valid pending-materialization state.

Performance constraint

Do not move Markdown materialization back onto the request/acceptance path.

The asynchronous path is producing the intended caller-facing improvement:

• SQLite acceptance throughput improved by approximately 31% at concurrency 8
• approximately 41% at concurrency 32
• approximately 54% at concurrency 64

The complete 100-note batch reached Markdown materialization approximately 24–30% later. That is an acceptable tradeoff: fast durable acceptance is the desired boundary, and file/search projections may catch up asynchronously.

The defect is the follow-up job's treatment of the valid pending state, not deferred materialization itself.

Existing recovery path is the inverse case

edit_note._resolve_after_disk_recovery() and POST /knowledge/index-file recover a Markdown file that already exists on disk but has not been indexed yet. This race is the opposite state: accepted NoteContent and the entity exist, while the Markdown projection is still pending. /index-file correctly returns 404 when the physical file does not exist, so that recovery path does not apply here.

Suggested direction

Prefer one of these state-aware approaches:

1. Reindex affected sources from their accepted NoteContent.markdown_content while materialization is pending; or
2. Explicitly defer/requeue the affected source reindex until materialization completes.

Avoid broadly swallowing FileNotFoundError, because a genuinely missing materialized file remains a correctness problem that should surface.

Cloud/local lifecycle parity should be preserved. Cloud already schedules accepted-note relation repair after materialization; local should not require the file before its corresponding materialization has completed.

Acceptance criteria

• write_note continues to acknowledge accepted mutations without awaiting Markdown materialization.
• A deliberately delayed materialization does not produce a background FileNotFoundError during relation resolution.
• Inbound forward references are still back-resolved after the target is accepted.
• The affected source search row is eventually refreshed with the accepted content and resolved relation metadata.
• Genuine missing-file inconsistencies are not silently swallowed.
• Regression coverage exercises an accepted entity whose Markdown file does not yet exist when relation resolution runs.
• Existing write-load acceptance performance is preserved.

Related

• [BUG] Forward references are not back-resolved when their target note is created (only on reindex) #1015 added the post-write relation-resolution follow-up so write_note could back-resolve inbound forward references. This issue tracks ordering that follow-up correctly around asynchronous materialization; it does not reopen the original missing-follow-up bug.
• DB-first note writes omit observations and relations until a later file re-index #1076 covered atomic graph persistence for DB-first accepted writes. This issue is specifically about the later search reindex reading the physical Markdown projection too early.

[claude]

Claude encountered an error after 0s —— View job

---

I'll analyze this and get back to you.