Skip to content

fix(provider): skip empty reasoning_content to preserve KV cache hits#28352

Open
nilo85 wants to merge 2 commits into
anomalyco:devfrom
nilo85:feature/fix-reasoning-content-cache
Open

fix(provider): skip empty reasoning_content to preserve KV cache hits#28352
nilo85 wants to merge 2 commits into
anomalyco:devfrom
nilo85:feature/fix-reasoning-content-cache

Conversation

@nilo85
Copy link
Copy Markdown

@nilo85 nilo85 commented May 19, 2026 •
edited
Loading

Issue for this PR

Closes #19081

Type of change

  • Bug fix

What does this PR do?

Historical assistant messages without reasoning were getting reasoning_content: "" forwarded to the LLM API. An empty string changes the token stream, breaking KV cache prefix matching — observed as 0% cache hits on 196K token prompts in production captures.

The interleaved reasoning block in normalizeMessages always set providerOptions.openaiCompatible[field] to reasoningText even when empty. The fix adds a guard so the field is only set when there's actual reasoning text. Messages without reasoning now keep the field absent, preserving KV cache prefix matching.

How did you verify your code works?

  • Unit tests added (226 pass) covering empty reasoning and no-reasoning cases
  • Direct API test against llama-swap: WITH reasoning_content → 97.2% cache hit, WITHOUT → 0% cache hit
  • Multi-turn KV cache test: Turn 1 (0%), Turn 2 (52.6%), Turn 3 (91.0%)

Screenshots / recordings

N/A — backend-only change, no UI.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

nilo85 added 2 commits May 19, 2026 15:56
@nilo85
fix(provider): skip empty reasoning_content to preserve KV cache hits
134bb34 
Historical assistant messages without reasoning were getting
reasoning_content: "" forwarded to the LLM API, breaking KV cache
prefix matching (0% cache hits on 196K token prompts).

Only set the interleaved field when reasoningText is non-empty.
@nilo85
test(provider): cover empty reasoning_content is not forwarded
17f2138 
Adds two tests:
- empty reasoning part does not set reasoning_content field
- assistant without reasoning parts keeps message unchanged
@github-actions github-actions Bot added needs:compliance This means the issue will auto-close after 2 hours. needs:issue labels May 19, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 19, 2026

Thanks for your contribution!

This PR doesn't have a linked issue. All PRs must reference an existing issue.

Please:

  1. Open an issue describing the bug/feature (if one doesn't exist)
  2. Add Fixes #<number> or Closes #<number> to this PR description

See CONTRIBUTING.md for details.

@nilo85 nilo85 marked this pull request as ready for review May 19, 2026 14:24
@github-actions github-actions Bot removed needs:compliance This means the issue will auto-close after 2 hours. needs:issue labels May 19, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 19, 2026

Thanks for updating your PR! It now meets our contributing guidelines. 👍

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 19, 2026

The following comment was made by an LLM, it may be inaccurate:

Potential Duplicate Found:

PR #28346: fix(llm): forward reasoning_content in experimental OpenAI Chat assistant messages
#28346

Why it's related: This PR directly addresses the inverse scenario of the current PR. While #28352 fixes the issue of empty reasoning_content breaking cache by skipping it, #28346 involves forwarding reasoning_content in assistant messages. These PRs are closely related and likely touch the same code paths in message normalization and provider handling. Verify these aren't addressing conflicting concerns or if one supersedes the other.

@nilo85
Copy link
Copy Markdown
Author

nilo85 commented May 19, 2026

image

Both fixes should be relevant as they cover same issue but different paths to get there

This was referenced May 19, 2026
Open
Open
Draft
nilo85 added a commit to nilo85/opencode that referenced this pull request May 19, 2026
@nilo85
feat(provider): detect and warn on history rewrite breaking KV cache
a9335bf 
Add detection for when messages sent to the LLM differ between turns,
which breaks KV cache prefix matching. Logs a warning with a formatted
diff showing what changed and why.

Catches issues like empty reasoning_content being added to messages
that previously had none (PR anomalyco#28352), as well as any other message
mutations during transformation.
nilo85 added a commit to nilo85/opencode that referenced this pull request May 19, 2026
@nilo85
feat(provider): detect and warn on history rewrite breaking KV cache
679d172 
Add detection for when messages sent to the LLM differ between turns,
which breaks KV cache prefix matching. Logs a warning with a formatted
diff showing what changed and why.

Catches issues like empty reasoning_content being added to messages
that previously had none (PR anomalyco#28352), as well as any other message
mutations during transformation.
nilo85 added a commit to nilo85/opencode that referenced this pull request May 19, 2026
@nilo85
feat(provider): detect and warn on history rewrite breaking KV cache
ed7e36c 
Add detection for when messages sent to the LLM differ between turns,
which breaks KV cache prefix matching. Logs a warning with a formatted
diff showing what changed and why.

Catches issues like empty reasoning_content being added to messages
that previously had none (PR anomalyco#28352), as well as any other message
mutations during transformation.
nilo85 added a commit to nilo85/opencode that referenced this pull request May 19, 2026
@nilo85
feat(provider): detect and warn on history rewrite breaking KV cache
f8e8298 
Add detection for when messages sent to the LLM differ between turns,
which breaks KV cache prefix matching. Logs a warning with a formatted
diff showing what changed and why.

Catches issues like empty reasoning_content being added to messages
that previously had none (PR anomalyco#28352), as well as any other message
mutations during transformation.
nilo85 added a commit to nilo85/opencode that referenced this pull request May 19, 2026
@nilo85
feat(provider): detect and warn on history rewrite breaking KV cache
c002206 
Add detection for when messages sent to the LLM differ between turns,
which breaks KV cache prefix matching. Logs a warning with a formatted
diff showing what changed and why.

Catches issues like empty reasoning_content being added to messages
that previously had none (PR anomalyco#28352), as well as any other message
mutations during transformation.
nilo85 added a commit to nilo85/opencode that referenced this pull request May 19, 2026
@nilo85
feat(provider): detect and warn on history rewrite breaking KV cache
e6dc518 
Add detection for when messages sent to the LLM differ between turns,
which breaks KV cache prefix matching. Logs a warning with a formatted
diff showing what changed and why.

Catches issues like empty reasoning_content being added to messages
that previously had none (PR anomalyco#28352), as well as any other message
mutations during transformation.
nilo85 added a commit to nilo85/opencode that referenced this pull request May 19, 2026
@nilo85
feat(provider): detect and warn on history rewrite breaking KV cache
563648b 
Add detection for when messages sent to the LLM differ between turns,
which breaks KV cache prefix matching. Logs a warning with a formatted
diff showing what changed and why.

Catches issues like empty reasoning_content being added to messages
that previously had none (PR anomalyco#28352), as well as any other message
mutations during transformation.
nilo85 added a commit to nilo85/opencode that referenced this pull request May 19, 2026
@nilo85
feat(provider): detect and warn on history rewrite breaking KV cache
2748d5f 
Add detection for when messages sent to the LLM differ between turns,
which breaks KV cache prefix matching. Logs a warning with a formatted
diff showing what changed and why.

Catches issues like empty reasoning_content being added to messages
that previously had none (PR anomalyco#28352), as well as any other message
mutations during transformation.
nilo85 added a commit to nilo85/opencode that referenced this pull request May 19, 2026
@nilo85
feat(provider): detect and warn on history rewrite breaking KV cache
b4ceb14 
Add detection for when messages sent to the LLM differ between turns,
which breaks KV cache prefix matching. Logs a warning with a formatted
diff showing what changed and why.

Catches issues like empty reasoning_content being added to messages
that previously had none (PR anomalyco#28352), as well as any other message
mutations during transformation.
nilo85 added a commit to nilo85/opencode that referenced this pull request May 19, 2026
@nilo85
feat(provider): detect and warn on history rewrite breaking KV cache
0dc861c 
Add detection for when messages sent to the LLM differ between turns,
which breaks KV cache prefix matching. Logs a warning with a formatted
diff showing what changed and why.

Catches issues like empty reasoning_content being added to messages
that previously had none (PR anomalyco#28352), as well as any other message
mutations during transformation.
@github-actions github-actions Bot mentioned this pull request May 20, 2026
Closed
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

reasoning_content stripped from assistant messages on replay, causing KV cache invalidation on local inference

1 participant

@nilo85