[BUG] Duplicate observation permalink collision drops the second observation from the search index

[phernandez]

Bug

Create one note with two genuinely different observations of the same category whose content is identical for the first 200 characters and differs only afterward (e.g. a 210-char shared prefix then '... ALPHA_UNIQUE_MARKER' vs '... BETA_UNIQUE_MARKER'). Searching for BETA_UNIQUE_MARKER (entity_types=['observation']) returns only ALPHA's row (matched via the shared prefix) — no result actually contains BETA_UNIQUE_MARKER. The BETA observation is silently missing from the search index and is unfindable.

Root cause

Observation.permalink (src/basic_memory/models/knowledge.py:255-258) truncates content to content[:200] before building the synthetic permalink {entity.permalink}/observations/{category}/{content[:200]}, so two observations identical in the first 200 chars produce identical permalinks. SearchService.index_entity_markdown (src/basic_memory/services/search_service.py:884-890) dedups rows via seen_permalinks and continues on a duplicate permalink, so the second observation row is never indexed.

Suggested fix

Make the indexed observation row identity unique even when synthetic permalinks collide. Options: (a) include the observation's DB id in the indexed permalink/id so distinct observations never collide; (b) include more (or a hash) of the full content in the synthetic permalink instead of a raw 200-char prefix; or (c) change the dedup key in index_entity_markdown from permalink to obs.id. Note the truncation exists for Postgres btree index limits, so prefer appending obs.id rather than removing truncation.

Repro (failing integration test, no mocks)

This test asserts the correct behavior and fails on current main (FAILS on HEAD: AssertionError - no result for query 'BETA_UNIQUE_MARKER' contains BETA_UNIQUE_MARKER; the single returned observation row is ALPHA's (matched on the shared 200-char prefix). Independen). It was produced by an automated integration-test bug hunt and can be dropped into test-int/ as the regression test once fixed.

"""Bughunt: distinct observations that share category + (first 200 chars of)
content collide on synthetic permalink and the second is silently dropped from
the search index, so it cannot be found via observation search.
"""

import json
from typing import Any

import pytest
from fastmcp import Client


def _json(tool_result) -> Any:
    assert len(tool_result.content) == 1
    assert tool_result.content[0].type == "text"
    return json.loads(tool_result.content[0].text)


@pytest.mark.asyncio
async def test_duplicate_category_content_observations_both_searchable(
    mcp_server, app, test_project
):
    async with Client(mcp_server) as client:
        prefix = "x" * 210  # > 200 so truncation makes them identical at [:200]
        content = (
            "# Dup Obs Note\n\n"
            "## Observations\n"
            f"- [note] {prefix} ALPHA_UNIQUE_MARKER\n"
            f"- [note] {prefix} BETA_UNIQUE_MARKER\n"
        )
        await client.call_tool(
            "write_note",
            {
                "project": test_project.name,
                "title": "Dup Obs Note",
                "directory": "dup",
                "content": content,
            },
        )
        beta = await client.call_tool(
            "search_notes",
            {
                "project": test_project.name,
                "query": "BETA_UNIQUE_MARKER",
                "search_type": "text",
                "entity_types": ["observation"],
                "output_format": "json",
            },
        )
        beta_data = _json(beta)
        snippets = [r.get("content") or "" for r in beta_data["results"]]
        assert any("BETA_UNIQUE_MARKER" in s for s in snippets), (
            "the second distinct observation (BETA) was dropped from the search index "
            "due to a synthetic-permalink collision (content truncated to 200 chars). "
            "No result contains BETA_UNIQUE_MARKER. results="
            + json.dumps(beta_data, default=str)[:800]
        )

---

🤖 Found via automated integration-test bug hunt (Claude Code)

[claude]

Claude finished @phernandez's task in 6m 15s —— View job

---

Triage & Fix

Classification: Bug | Priority: High | Component: database/search | Complexity: Medium

Labels added: bug

• Analyze issue and classify
• Read relevant source files (knowledge.py, search_service.py)
• Implement fix in search_service.py
• Add regression test
• Commit changes (commit 32f3858)
• Push to remote — blocked by 403 (workflow token lacks write access)

Root Cause Confirmed

In src/basic_memory/services/search_service.py:884, the seen_permalinks set was the dedup key for observations during search indexing. Observation.permalink truncates content to 200 chars (Postgres btree limit), so two observations with the same category and identical 200-char prefix produce the same synthetic permalink. The second observation was silently skipped via continue.

Fix

src/basic_memory/services/search_service.py — dedup by obs.id (always unique DB primary key) instead of the synthetic permalink. Colliding indexed permalinks are disambiguated by appending obs.id:

# Before (buggy) — second obs silently dropped when permalinks collide
seen_permalinks: set[str] = {entity.permalink} if entity.permalink else set()
for obs in entity.observations:
    obs_permalink = obs.permalink
    if obs_permalink in seen_permalinks:
        continue
    seen_permalinks.add(obs_permalink)

# After (fixed)
seen_obs_ids: set[int] = set()
seen_indexed_permalinks: set[str] = {entity.permalink} if entity.permalink else set()
for obs in entity.observations:
    if obs.id in seen_obs_ids:
        continue
    seen_obs_ids.add(obs.id)
    obs_permalink = obs.permalink
    if obs_permalink in seen_indexed_permalinks:
        obs_permalink = f"{obs_permalink}/{obs.id}"  # disambiguate for navigation
    seen_indexed_permalinks.add(obs_permalink)

test-int/mcp/test_observation_permalink_collision_integration.py — regression test: two observations with 210-char shared prefix (> 200 truncation point), asserting both ALPHA and BETA are independently searchable.

Push Blocked

The push script returned remote: Permission to basicmachines-co/basic-memory.git denied to github-actions[bot] (HTTP 403). The workflow token doesn't have write access. A maintainer can apply the changes manually — the diff is:

Patch for src/basic_memory/services/search_service.py

Replace lines 884-890 with:

            # Dedup by obs.id (the DB primary key) rather than by synthetic permalink.
            # Trigger: two observations with the same category and identical first 200
            # chars of content produce the same synthetic permalink (Postgres btree limit
            # forces content[:200]).
            # Why: the second observation was silently skipped, making it unfindable.
            # Outcome: every distinct observation gets an index row; colliding permalinks
            # are disambiguated by appending the obs id so navigation still works.
            seen_obs_ids: set[int] = set()
            seen_indexed_permalinks: set[str] = {entity.permalink} if entity.permalink else set()
            for obs in entity.observations:
                if obs.id in seen_obs_ids:
                    logger.debug(f"Skipping duplicate observation id: {obs.id}")
                    continue
                seen_obs_ids.add(obs.id)

                obs_permalink = obs.permalink
                # Append obs.id when the synthetic permalink collides so the indexed
                # permalink is always unique (entity vs obs, or obs vs obs with same prefix).
                if obs_permalink in seen_indexed_permalinks:
                    obs_permalink = f"{obs_permalink}/{obs.id}"
                seen_indexed_permalinks.add(obs_permalink)