Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 27 additions & 2 deletions docs/semantic-search.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,8 +117,8 @@ All settings are fields on `BasicMemoryConfig` and can be set via environment va
| `milvus_database` | `BASIC_MEMORY_MILVUS_DATABASE` | `"default"` | Milvus database name. |
| `semantic_embedding_provider` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER` | `"fastembed"` | Embedding provider: `"fastembed"` (local), `"openai"` (API), or `"litellm"` (multi-provider API, **experimental** — advanced users only). |
| `semantic_embedding_model` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_MODEL` | `"bge-small-en-v1.5"` | Model identifier. FastEmbed models must exist in the installed FastEmbed catalog. Auto-adjusted per provider if left at default. |
| `semantic_embedding_api_base` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_API_BASE` | Unset | Optional custom endpoint for the LiteLLM provider, including local or self-hosted OpenAI-compatible servers. |
| `semantic_embedding_api_key` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_API_KEY` | Unset | Optional API key passed directly to the LiteLLM provider. When unset, LiteLLM continues to read provider credential env vars such as `OPENAI_API_KEY`. |
| `semantic_embedding_api_base` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_API_BASE` | Unset | Optional custom endpoint for the `openai` or `litellm` provider — point either at a local or self-hosted OpenAI-compatible server. |
| `semantic_embedding_api_key` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_API_KEY` | Unset | Optional API key for the `openai` or `litellm` provider. When unset, `openai` falls back to `OPENAI_API_KEY` and `litellm` reads its provider credential env vars. |
| `semantic_embedding_dimensions` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_DIMENSIONS` | Provider default | Vector dimensions. Defaults to 384 for FastEmbed and 1536 for OpenAI/LiteLLM OpenAI. Set this to the model's output size when choosing a non-default FastEmbed or LiteLLM model. |
| `semantic_embedding_forward_dimensions` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_FORWARD_DIMENSIONS` | Auto | LiteLLM-only override for whether configured dimensions are sent as a provider-side output-size request. |
| `semantic_embedding_batch_size` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_BATCH_SIZE` | `2` | Number of texts to embed per batch. |
Expand Down Expand Up @@ -229,6 +229,31 @@ export BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER=openai
export OPENAI_API_KEY=sk-...
```

#### OpenAI-compatible endpoints (local or self-hosted)

The `openai` provider also honors `semantic_embedding_api_base` and
`semantic_embedding_api_key`, so you can point it at any OpenAI-compatible
embedding server — llama.cpp (`llama-server --embedding`), vLLM, Text Embeddings
Inference, LM Studio, or Ollama's OpenAI shim — without moving to the
experimental LiteLLM provider:

```bash
export BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED=true
export BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER=openai
export BASIC_MEMORY_SEMANTIC_EMBEDDING_MODEL=your-model
export BASIC_MEMORY_SEMANTIC_EMBEDDING_API_BASE=http://localhost:8080/v1
# Set the key your server expects; local servers often ignore it, but the
# OpenAI client still requires a non-empty value.
export BASIC_MEMORY_SEMANTIC_EMBEDDING_API_KEY=sk-local
export BASIC_MEMORY_SEMANTIC_EMBEDDING_DIMENSIONS=<model output size>
bm reindex --embeddings
```

Set `semantic_embedding_dimensions` to the model's output size — Basic Memory
creates fixed-dimension vector storage before indexing. The key is optional only
when the server accepts requests without one and `OPENAI_API_KEY` is set;
otherwise provide it explicitly.

### LiteLLM

> **Experimental — advanced users only.** The LiteLLM provider is experimental and aimed at users comfortable operating remote embedding backends: paid API calls, per-model dimension and input-role configuration, and slower reindexing of large corpora. For most users, FastEmbed (local, default) is recommended. See [LiteLLM Provider](litellm-provider.md) for the caveats and tuning.
Expand Down
11 changes: 6 additions & 5 deletions src/basic_memory/config_models.py
Original file line number Diff line number Diff line change
Expand Up @@ -300,16 +300,17 @@ def __init__(self, **data: Any) -> None: ...
semantic_embedding_api_base: str | None = Field(
default=None,
description=(
"Optional custom API base URL for the LiteLLM embedding provider. "
"Use this for OpenAI-compatible local or self-hosted embedding servers."
"Optional custom API base URL for the openai or litellm embedding "
"provider. Use this for OpenAI-compatible local or self-hosted "
"embedding servers (llama.cpp, vLLM, TEI, LM Studio, Ollama)."
),
)
semantic_embedding_api_key: str | None = Field(
default=None,
description=(
"Optional API key passed directly to the LiteLLM embedding provider. "
"When unset, LiteLLM continues to resolve credentials from provider "
"environment variables such as OPENAI_API_KEY."
"Optional API key passed directly to the openai or litellm embedding "
"provider. When unset, the openai provider falls back to OPENAI_API_KEY "
"and litellm resolves provider credential environment variables."
),
)
semantic_embedding_dimensions: int | None = Field(
Expand Down
16 changes: 9 additions & 7 deletions src/basic_memory/repository/embedding_provider_factory.py
Original file line number Diff line number Diff line change
Expand Up @@ -116,17 +116,17 @@ def _provider_cache_key(app_config: BasicMemoryConfig) -> ProviderCacheKey:
runtime CPU budget makes the key drift between calls in a container (#872).
"""
provider_name = app_config.semantic_embedding_provider.strip().lower()
litellm_api_base_digest = None
litellm_api_key_digest = None
if provider_name == "litellm":
litellm_api_base_digest = _sensitive_value_digest(app_config.semantic_embedding_api_base)
litellm_api_key_digest = _sensitive_value_digest(app_config.semantic_embedding_api_key)
api_base_digest = None
api_key_digest = None
if provider_name in {"openai", "litellm"}:
api_base_digest = _sensitive_value_digest(app_config.semantic_embedding_api_base)
api_key_digest = _sensitive_value_digest(app_config.semantic_embedding_api_key)

return (
provider_name,
app_config.semantic_embedding_model,
litellm_api_base_digest,
litellm_api_key_digest,
api_base_digest,
api_key_digest,
app_config.semantic_embedding_dimensions,
app_config.semantic_embedding_forward_dimensions,
app_config.semantic_embedding_batch_size,
Expand Down Expand Up @@ -266,6 +266,8 @@ def create_embedding_provider(app_config: BasicMemoryConfig) -> EmbeddingProvide
model_name = "text-embedding-3-small"
provider = OpenAIEmbeddingProvider(
model_name=model_name,
api_key=app_config.semantic_embedding_api_key,
base_url=app_config.semantic_embedding_api_base,
Comment thread
phernandez marked this conversation as resolved.
batch_size=app_config.semantic_embedding_batch_size,
request_concurrency=app_config.semantic_embedding_request_concurrency,
**extra_kwargs,
Expand Down
42 changes: 42 additions & 0 deletions tests/repository/test_openai_provider.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
from basic_memory.config import BasicMemoryConfig
import basic_memory.repository.embedding_provider_factory as embedding_provider_factory_module
from basic_memory.repository.embedding_provider_factory import (
_provider_cache_key,
create_embedding_provider,
reset_embedding_provider_cache,
)
Expand Down Expand Up @@ -179,6 +180,47 @@ def test_embedding_provider_factory_selects_openai_and_applies_default_model():
assert provider.model_name == "text-embedding-3-small"


def test_embedding_provider_factory_forwards_openai_api_configuration():
config = BasicMemoryConfig(
env="test",
projects={"test-project": "/tmp/basic-memory-test"},
default_project="test-project",
semantic_search_enabled=True,
semantic_embedding_provider="openai",
semantic_embedding_api_base="https://embedding.example/v1",
semantic_embedding_api_key="test-key",
)

provider = create_embedding_provider(config)

assert isinstance(provider, OpenAIEmbeddingProvider)
assert provider._base_url == "https://embedding.example/v1"
assert provider._api_key == "test-key"


def test_embedding_provider_factory_separates_openai_api_cache_keys():
base = dict(
env="test",
projects={"test-project": "/tmp/basic-memory-test"},
default_project="test-project",
semantic_search_enabled=True,
semantic_embedding_provider="openai",
semantic_embedding_api_base="https://one.example/v1",
semantic_embedding_api_key="test-key",
)

first = _provider_cache_key(BasicMemoryConfig(**base))
second = _provider_cache_key(
BasicMemoryConfig(**{**base, "semantic_embedding_api_base": "https://two.example/v1"})
)
third = _provider_cache_key(
BasicMemoryConfig(**{**base, "semantic_embedding_api_key": "other-key"})
)

assert first != second
assert first != third


def test_embedding_provider_factory_rejects_unknown_provider():
"""Factory should fail fast for unsupported provider names."""
config = BasicMemoryConfig(
Expand Down
Loading