Repository navigation
fix(models): send structured output for non-BaseModel schemas in LiteLlm - #7948
Open
sumedh-aerram wants to merge 1 commit into
Open
sumedh-aerram wants to merge 1 commit into
sumedh-aerram wants to merge 1 commit into
Conversation
_to_litellm_response_format only handled dicts and BaseModels, so list, int, Literal and Annotated output schemas got no response_format, and Annotated aliases raised a TypeError. Convert them with schema_to_json_schema. For strict json_schema targets, nest non-object roots under a "response" property, keeping $defs at the root, and unwrap the value from the reply, including the final streaming aggregate. Unwrap only when ADK wrapped. Map schemas and unconvertible types keep the previous warning. Fall back to response_json_schema when response_schema is unset. The Gemini path is unchanged. Fixes google#7946
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please ensure you have read the contribution guide before creating a pull request.
Link to Issue or Description of Change
1. Link to an existing issue (if applicable):
Problem:
_to_litellm_response_formatonly converted dicts andBaseModelclasses or instances. For any otheroutput_schema(list[Item],int,Literal[...],Annotated[...]) it loggedUnsupported response_schema type ... for LiteLLM structured outputsand returnedNone, so noresponse_formatwas sent and the model answered unconstrained.Annotated[Model, Field(...)]actually raised aTypeError, because the alias passed themodel_dumpcheck.Solution:
schema_to_json_schema.json_schematargets, which require an object root, a non-object root is nested as{"type": "object", "properties": {"response": <schema>}, "required": ["response"], "additionalProperties": false}, with$defskept at the document root. This mirrors the native Anthropic adapter's structured-output tool.json.dumps(reply["response"])), for non-streaming responses and for the final streaming aggregate.BaseModelreply that contains aresponsekey is untouched. So is a reply when the caller passed their ownresponse_format.json_object+response_schema) is unchanged and is never wrapped.dict[str, int], an object with only a schema inadditionalProperties) can't be expressed in OpenAI strict mode. They keep the previous behaviour: a warning and noresponse_format. Types thatschema_to_json_schemacan't convert are handled the same way.config.response_schemais unset,config.response_json_schemais used. That's howLiteral/Annotatedoutput schemas arrive after fix(agents): keep Literal and Annotated output schemas #7928, and how callers set a JSON schema directly._to_litellm_response_formatkeeps its signature._enforce_strict_openai_schemais not modified (it is touched by fix(openai): keep $defs of a recursive output_schema in strict mode #7806).Testing Plan
Unit Tests:
New tests in
tests/unittests/models/test_litellm.pycover:response_formatforlist[M],int,dict[str, int],LiteralandAnnotatedon OpenAI and Gemini model strings;response_json_schemafallback;BaseModelreply with aresponsekey, and a callerresponse_formatoverride. Each fails if its guard is removed;The new regression cases fail on
mainand pass with the change.macOS:
pytest tests/unittests/models: 1959 passed on each of Python 3.11, 3.12, 3.13 and 3.14.pytest tests/unittests -n autoon Python 3.11 (sandboxed runner, before the final test-only commit): 22672 passed, 46 failed, 3 errors. Every failure and error ID also occurs without this change: telemetry goldens, A2A, macOS VMaaS, and sandbox restrictions on sockets/file writes. No new failures.pre-commiton the changed files: clean.mypyonlite_llm.py: no new errors.Manual End-to-End (E2E) Tests:
Live against OpenAI (
LiteLlm(model="openai/gpt-4o-mini"), viaLlmRequest.set_output_schema+generate_content_async):output_schemamainlist[Item][{"name": "Vacuum Cleaner"}, {"name": "Toaster"}], validates1. Toaster\n2. blender, fails validationLiteral["yes", "no"]"yes", validatesYes., fails validationint8, validatesA spider has eight legs., fails validationItem(BaseModel){"name":"Coffee Mug"}, validates{"name":"Toaster"}, validates (unchanged)Checklist
Additional context
json_schema(for exampleanthropic/,ollama_chat/,azure/) get the same wrapping as OpenAI, since they also take the strict object-root format. They were checked by request inspection only, not live.response_formaton strict targets, as before.