Skip to content

fix(models): send structured output for non-BaseModel schemas in LiteLlm - #7948

Open
sumedh-aerram wants to merge 1 commit into
google:mainfrom
sumedh-aerram:fix/litellm-non-basemodel-output-schema
Open

sumedh-aerram wants to merge 1 commit into
google:mainfrom
sumedh-aerram:fix/litellm-non-basemodel-output-schema

Conversation

@sumedh-aerram

Copy link
Copy Markdown

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

Problem:
_to_litellm_response_format only converted dicts and BaseModel classes or instances. For any other output_schema (list[Item], int, Literal[...], Annotated[...]) it logged Unsupported response_schema type ... for LiteLLM structured outputs and returned None, so no response_format was sent and the model answered unconstrained. Annotated[Model, Field(...)] actually raised a TypeError, because the alias passed the model_dump check.

Solution:

  • Remaining schema types are converted with the existing schema_to_json_schema.
  • For OpenAI-style strict json_schema targets, which require an object root, a non-object root is nested as {"type": "object", "properties": {"response": <schema>}, "required": ["response"], "additionalProperties": false}, with $defs kept at the document root. This mirrors the native Anthropic adapter's structured-output tool.
  • The reply is unwrapped back to the value (json.dumps(reply["response"])), for non-streaming responses and for the final streaming aggregate.
  • Unwrapping happens only when ADK did the wrapping. A BaseModel reply that contains a response key is untouched. So is a reply when the caller passed their own response_format.
  • Partial streaming chunks keep the raw wrapped text.
  • LiteLLM's Gemini path (json_object + response_schema) is unchanged and is never wrapped.
  • Map schemas (dict[str, int], an object with only a schema in additionalProperties) can't be expressed in OpenAI strict mode. They keep the previous behaviour: a warning and no response_format. Types that schema_to_json_schema can't convert are handled the same way.
  • When config.response_schema is unset, config.response_json_schema is used. That's how Literal/Annotated output schemas arrive after fix(agents): keep Literal and Annotated output schemas #7928, and how callers set a JSON schema directly.
  • _to_litellm_response_format keeps its signature. _enforce_strict_openai_schema is not modified (it is touched by fix(openai): keep $defs of a recursive output_schema in strict mode #7806).

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally. (See below: pre-existing environment-dependent failures only.)

New tests in tests/unittests/models/test_litellm.py cover:

  • the response_format for list[M], int, dict[str, int], Literal and Annotated on OpenAI and Gemini model strings;
  • unwrapping of wrapped replies, streaming and non-streaming;
  • the response_json_schema fallback;
  • the unsupported-type warning;
  • guard tests: a BaseModel reply with a response key, and a caller response_format override. Each fails if its guard is removed;
  • thought parts and non-ASCII text in wrapped replies.

The new regression cases fail on main and pass with the change.

macOS:

  • pytest tests/unittests/models: 1959 passed on each of Python 3.11, 3.12, 3.13 and 3.14.
  • Full pytest tests/unittests -n auto on Python 3.11 (sandboxed runner, before the final test-only commit): 22672 passed, 46 failed, 3 errors. Every failure and error ID also occurs without this change: telemetry goldens, A2A, macOS VMaaS, and sandbox restrictions on sockets/file writes. No new failures.
  • pre-commit on the changed files: clean. mypy on lite_llm.py: no new errors.

Manual End-to-End (E2E) Tests:

Live against OpenAI (LiteLlm(model="openai/gpt-4o-mini"), via LlmRequest.set_output_schema + generate_content_async):

output_schema This PR main
list[Item] [{"name": "Vacuum Cleaner"}, {"name": "Toaster"}], validates 1. Toaster\n2. blender, fails validation
Literal["yes", "no"] "yes", validates Yes., fails validation
int 8, validates A spider has eight legs., fails validation
Item (BaseModel) {"name":"Coffee Mug"}, validates {"name":"Toaster"}, validates (unchanged)

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes. (New and affected tests pass; see the pre-existing failures above.)
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

Additional context

  • Other LiteLLM providers that receive json_schema (for example anthropic/, ollama_chat/, azure/) get the same wrapping as OpenAI, since they also take the strict object-root format. They were checked by request inspection only, not live.
  • Map schemas still get no response_format on strict targets, as before.

_to_litellm_response_format only handled dicts and BaseModels, so list,
int, Literal and Annotated output schemas got no response_format, and
Annotated aliases raised a TypeError.

Convert them with schema_to_json_schema. For strict json_schema targets,
nest non-object roots under a "response" property, keeping $defs at the
root, and unwrap the value from the reply, including the final streaming
aggregate. Unwrap only when ADK wrapped. Map schemas and unconvertible
types keep the previous warning. Fall back to response_json_schema when
response_schema is unset. The Gemini path is unchanged.

Fixes google#7946
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LiteLlm sends no response_format for non-BaseModel output schemas (list, int, dict, Literal)

2 participants