Binary tool results not passed to ExternalToolTextResultForLlm

[sophiehenning]

In

copilot-sdk/python/copilot/session.py

Line 1649 in 1600b57

result=ExternalToolTextResultForLlm(

, an ExternalToolTextResultForLlm is created to be sent to HandlePendingToolCallRequest. However, the binary_results_for_llm attribute from tool_result is not copied over, but only the text result.
I believe this could be the reason why the agent I am building is unable to process image results of tools (stating that it cannot view the image, while directly sending an image as a blob attachment works fine).

[github-actions]

Investigation findings: confirmed bug ✅

I investigated the reported behavior and can confirm this is a genuine bug.

What I looked at

• python/copilot/tools.py — ToolResult dataclass definition
• python/copilot/session.py — _execute_tool_and_respond method (around line 1649)
• python/copilot/generated/rpc.py — ExternalToolTextResultForLlm class definition

Root cause

In session.py, when building the ExternalToolTextResultForLlm to send via handle_pending_tool_call, only four fields are copied from the ToolResult:

result=ExternalToolTextResultForLlm(
    text_result_for_llm=tool_result.text_result_for_llm,
    error=tool_result.error,
    result_type=tool_result.result_type,
    tool_telemetry=tool_result.tool_telemetry,
),

Two fields from ToolResult are silently dropped:

Dropped field	ToolResult type	ExternalToolTextResultForLlm type
binary_results_for_llm	list[ToolBinaryResult] | None	list[ExternalToolTextResultForLlmBinaryResultsForLlm] | None
session_log	str | None	str | None

binary_results_for_llm is the directly relevant omission for this issue — it means any base64-encoded images or binary data returned by a tool are never forwarded to the model. The session_log field is similarly lost.

Note that a type conversion is also needed for binary_results_for_llm: ToolBinaryResult.type is a plain string literal "image" | "resource", while ExternalToolTextResultForLlmBinaryResultsForLlm.type expects the ExternalToolTextResultForLlmBinaryResultsForLlmType enum.

Why this affects you

Your observation is correct — directly attaching a binary blob works because it bypasses this code path entirely. But when a tool returns a ToolResult with binary_results_for_llm populated, those binaries are silently discarded here, so the model never receives them and correctly reports it "cannot view the image."

Generated by Bug Handler for issue #1644 · sonnet46 1.1M · ◷

[briandunnington]

seeing this same behavior in C#/.NET