Skip to content

LocalEvalService.evaluate adds rubrics to the caller's inference results, so scoring them again raises #7874

Description

@adityasingh2400

🔴 Required Information

Describe the Bug:
LocalEvalService.evaluate modifies the inference results it is given. If you score the same results twice, the second call raises and aborts the whole run.

_evaluate_single_inference_result copies the eval case's rubrics, and each expected invocation's rubrics, onto inference_result.inferences. Those are the caller's own Invocation objects. A second evaluate over them finds the rubric ids already present, and _add_rubrics_to_invocation raises ValueError: Rubric with rubric_id '...' already exists.

That happens outside the per-metric error handling, so nothing is scored or saved. It happens for any metric, even ones that ignore rubrics such as response_match_score. Results serialized after a first evaluate carry the rubrics too, so they fail the same way.

This breaks the flow in the eval service guide's "Score a run again without rerunning it" section: keep a run and score it again later against a stricter threshold or an extra metric.

Steps to Reproduce:

  1. Add an eval case with a case-level rubric, then run perform_inference on it.
  2. Call evaluate on the inference results, with any metric.
  3. Call evaluate on the same results again, e.g. with a stricter threshold.

Expected Behavior:
Both evaluate calls score the run, and the caller's inference results come back unchanged.

Observed Behavior:

1st evaluate: PASSED | caller's inference now carries rubrics: ['names_device']
2nd evaluate (same objects) raised: ValueError Rubric with rubric_id 'names_device' already exists.
evaluate on results serialized BEFORE first evaluate: PASSED
evaluate on results serialized AFTER first evaluate raised: ValueError Rubric with rubric_id 'names_device' already exists.

Environment Details:

  • ADK Library Version (pip show google-adk): 2.13.0 (also on main at e9e8f61)
  • Desktop OS: macOS
  • Python Version (python -V): 3.11.16

Model Information:

  • Are you using LiteLLM: No
  • Which model is being used: any (the repro uses a fake BaseLlm; scoring with response_match_score needs no model)

🟡 Optional Information

Minimal Reproduction Code:

mgr = InMemoryEvalSetsManager()
mgr.create_eval_set(app_name="app", eval_set_id="s1")
mgr.add_eval_case(app_name="app", eval_set_id="s1", eval_case=EvalCase(
    eval_id="c1",
    conversation=[Invocation(
        user_content=types.Content(role="user", parts=[types.Part(text="Turn off the bedroom light.")]),
        final_response=types.Content(role="model", parts=[types.Part(text="The bedroom light is off.")]))],
    rubrics=[Rubric(rubric_id="names_device", rubric_content=RubricContent(text_property="Names the device."))]))
svc = LocalEvalService(root_agent=agent, eval_sets_manager=mgr)

infs = [r async for r in svc.perform_inference(
    InferenceRequest(app_name="app", eval_set_id="s1", inference_config=InferenceConfig()))]
loose = EvaluateConfig(eval_metrics=[EvalMetric(metric_name="response_match_score", threshold=0.5)])
strict = EvaluateConfig(eval_metrics=[EvalMetric(metric_name="response_match_score", threshold=0.9)])

[r async for r in svc.evaluate(EvaluateRequest(inference_results=infs, evaluate_config=loose))]
[r async for r in svc.evaluate(EvaluateRequest(inference_results=infs, evaluate_config=strict))]  # ValueError

How often has this issue occurred?:

  • Always (100%), when the eval case or its expected invocations have rubrics
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

eval[Component] This issue is related to evaluation

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions