Repository navigation
Description
🔴 Required Information
Describe the Bug:
LocalEvalService.evaluate modifies the inference results it is given. If you score the same results twice, the second call raises and aborts the whole run.
_evaluate_single_inference_result copies the eval case's rubrics, and each expected invocation's rubrics, onto inference_result.inferences. Those are the caller's own Invocation objects. A second evaluate over them finds the rubric ids already present, and _add_rubrics_to_invocation raises ValueError: Rubric with rubric_id '...' already exists.
That happens outside the per-metric error handling, so nothing is scored or saved. It happens for any metric, even ones that ignore rubrics such as response_match_score. Results serialized after a first evaluate carry the rubrics too, so they fail the same way.
This breaks the flow in the eval service guide's "Score a run again without rerunning it" section: keep a run and score it again later against a stricter threshold or an extra metric.
Steps to Reproduce:
- Add an eval case with a case-level rubric, then run
perform_inferenceon it. - Call
evaluateon the inference results, with any metric. - Call
evaluateon the same results again, e.g. with a stricter threshold.
Expected Behavior:
Both evaluate calls score the run, and the caller's inference results come back unchanged.
Observed Behavior:
1st evaluate: PASSED | caller's inference now carries rubrics: ['names_device']
2nd evaluate (same objects) raised: ValueError Rubric with rubric_id 'names_device' already exists.
evaluate on results serialized BEFORE first evaluate: PASSED
evaluate on results serialized AFTER first evaluate raised: ValueError Rubric with rubric_id 'names_device' already exists.
Environment Details:
- ADK Library Version (pip show google-adk): 2.13.0 (also on
mainat e9e8f61) - Desktop OS: macOS
- Python Version (python -V): 3.11.16
Model Information:
- Are you using LiteLLM: No
- Which model is being used: any (the repro uses a fake
BaseLlm; scoring withresponse_match_scoreneeds no model)
🟡 Optional Information
Minimal Reproduction Code:
mgr = InMemoryEvalSetsManager()
mgr.create_eval_set(app_name="app", eval_set_id="s1")
mgr.add_eval_case(app_name="app", eval_set_id="s1", eval_case=EvalCase(
eval_id="c1",
conversation=[Invocation(
user_content=types.Content(role="user", parts=[types.Part(text="Turn off the bedroom light.")]),
final_response=types.Content(role="model", parts=[types.Part(text="The bedroom light is off.")]))],
rubrics=[Rubric(rubric_id="names_device", rubric_content=RubricContent(text_property="Names the device."))]))
svc = LocalEvalService(root_agent=agent, eval_sets_manager=mgr)
infs = [r async for r in svc.perform_inference(
InferenceRequest(app_name="app", eval_set_id="s1", inference_config=InferenceConfig()))]
loose = EvaluateConfig(eval_metrics=[EvalMetric(metric_name="response_match_score", threshold=0.5)])
strict = EvaluateConfig(eval_metrics=[EvalMetric(metric_name="response_match_score", threshold=0.9)])
[r async for r in svc.evaluate(EvaluateRequest(inference_results=infs, evaluate_config=loose))]
[r async for r in svc.evaluate(EvaluateRequest(inference_results=infs, evaluate_config=strict))] # ValueErrorHow often has this issue occurred?:
- Always (100%), when the eval case or its expected invocations have rubrics
Activity