PerTurnUserSimulatorQualityV1 scores unjudged turns as failures #7886

🔴 Required Information

Describe the Bug:
PerTurnUserSimulatorQualityV1 counts turns it could not judge as failures. One judge reply without a parseable is_valid turns a passing conversation into a failing one. A conversation where no turn could be judged reports 0.0 FAILED instead of NOT_EVALUATED.

_aggregate_conversation_results increments num_evaluated for every turn, including turns marked NOT_EVALUATED. A turn is marked that way when the judge reply is empty or unparseable, or when it is a first user turn with no text (#6405). Each such turn adds 0 to the score but 1 to the count. Because every turn is counted, the function's own num_evaluated == 0 branch, which returns NOT_EVALUATED, can never run.

FinalResponseMatchV2Evaluator and HallucinationsV1Evaluator already skip unscored results, and AgentEvaluator averages only scored turns. So the same results pass under pytest and fail under adk eval.

Steps to Reproduce:

1. Evaluate a conversation scenario with per_turn_user_simulator_quality_v1 and threshold 1.0.
2. Have the judge approve turn 1 and return a reply without is_valid for turn 2.

Expected Behavior:
The unjudged turn is left out, so the conversation scores 1.0 PASSED. A conversation with no judged turns is NOT_EVALUATED.

Observed Behavior:

PASSED (1.0) + NOT_EVALUATED  -> 0.5 FAILED
only NOT_EVALUATED turns      -> 0.0 FAILED

Environment Details:

• ADK Library Version (pip show google-adk): 2.13.0 (also on main at e9e8f61)
• Desktop OS: macOS
• Python Version (python -V): 3.11.16

Model Information:

• Are you using LiteLLM: No
• Which model is being used: Gemini (gemini-2.5-flash) as the judge

🟡 Optional Information

Minimal Reproduction Code:

from google.adk.evaluation.eval_case import Invocation
from google.adk.evaluation.eval_metrics import EvalMetric
from google.adk.evaluation.eval_metrics import JudgeModelOptions
from google.adk.evaluation.eval_metrics import LlmBackedUserSimulatorCriterion
from google.adk.evaluation.evaluator import EvalStatus
from google.adk.evaluation.evaluator import PerInvocationResult
from google.adk.evaluation.simulation.per_turn_user_simulator_quality_v1 import PerTurnUserSimulatorQualityV1
from google.genai import types

evaluator = PerTurnUserSimulatorQualityV1(
    EvalMetric(
        metric_name="per_turn_user_simulator_quality_v1",
        threshold=1.0,
        criterion=LlmBackedUserSimulatorCriterion(
            threshold=1.0,
            judge_model_options=JudgeModelOptions(judge_model="gemini-2.5-flash"),
        ),
    )
)


def turn(text, score, status):
  return PerInvocationResult(
      actual_invocation=Invocation(user_content=types.UserContent(text)),
      score=score,
      eval_status=status,
  )


# Turn 1 passed; the judge's reply for turn 2 had no parseable `is_valid`.
r = evaluator._aggregate_conversation_results([
    turn("hi", 1.0, EvalStatus.PASSED),
    turn("book it", None, EvalStatus.NOT_EVALUATED),
])
print(r.overall_score, r.overall_eval_status)  # 0.5 EvalStatus.FAILED

# No turn could be judged at all.
r = evaluator._aggregate_conversation_results([
    turn("hi", None, EvalStatus.NOT_EVALUATED),
])
print(r.overall_score, r.overall_eval_status)  # 0.0 EvalStatus.FAILED

How often has this issue occurred?:

• Always (100%), whenever a turn is not evaluated