Repository navigation
fix(metrics): report unscored references in the WER summary line - #3787
Conversation
The last line of the WER report, "Scored N sentences, M not present in hyp.", was computed from the hypothesis file: N was the number of hypothesis lines and M the number of hypotheses without a reference. A reference whose hypothesis is missing (for example an utterance the decoder dropped) was therefore never reported, and a stray hypothesis was counted as scored. With three references and only two of them decoded, the report read "Scored 2 sentences, 0 not present in hyp." Count the utterances actually scored and the references that have no hypothesis, matching the Kaldi compute-wer summary this line mirrors. Apply the same fix to the runtime and Aishell copies of compute_wer.py.
|
Correction to my earlier review: I incorrectly treated DCO as a mandatory gate for this PR. The current contribution guide, branch protection/rules and checks do not establish that requirement. I withdraw that request; no commit rewrite or sign-off is needed to satisfy the review requirement I invented. Sorry for the unnecessary delay. Exact head 4444e35, main 24217c8, synthetic merge a21aa21 are unchanged. The recorded 2026-10-22 03:37 UTC run had base+submitted tests 12 passed/5 failed; head and merge each 17 targeted plus 2 adjacent tests passed, with compileall and exact six-file tree checks passing. The receipt hashes and refs were checked again, without rerunning unchanged tests. This is Python 3.12 CPU report-summary validation, not GPU inference, hosted CI or full-suite evidence. |
LauraGPT
left a comment
LauraGPT
left a comment
There was a problem hiding this comment.
Independent review: the five summary implementations now count scored reference/hypothesis pairs and missing reference-side hypotheses correctly; stray hypotheses no longer inflate the summary. WER, SER and the existing skip behavior are unchanged. The fixture covers missing and stray hypotheses in the library and four copies. No correctness findings remain in this focused change. The earlier unsupported DCO gate has been withdrawn.
Exact head 4444e35, main 24217c8, synthetic merge a21aa21 are unchanged. The recorded 2026-10-22 03:37 UTC run had base+submitted tests 12 passed/5 failed; head and merge each 17 targeted plus 2 adjacent tests passed, with compileall and exact six-file tree checks passing. The receipt hashes and refs were checked again, without rerunning unchanged tests. This is Python 3.12 CPU report-summary validation, not GPU inference, hosted CI or full-suite evidence.
Summary
The last line of the WER report,
Scored N sentences, M not present in hyp., was computed from the hypothesis file:Nwas the number of hypothesis lines andMthe number of hypotheses without a reference. A reference whose hypothesis is missing (for example an utterance the decoder dropped) was never reported, and a stray hypothesis was counted as scored.This counts the utterances actually scored and the references that have no hypothesis, matching the Kaldi
compute-wersummary the line mirrors. The same two-line fix is applied tofunasr/metrics/wer.pyand the four standalone copies (runtime/python/utils,runtime/python/libtorch/funasr_torch/utils,examples/aishell/paraformer/utils,examples/aishell/e_paraformer/utils).Repro: ref has
utt1,utt2,utt3; hyp hasutt1andutt3only.Before:
After:
With an extra hypothesis
utt9that has no reference, the old line readScored 3 sentences, 1 not present in hyp.; it now readsScored 2 sentences, 1 not present in hyp.Type of change
Validation
python -m compileall funasr examples testsDocs or links checked
Runtime/deployment command tested
pytest tests/test_metrics_wer.py: 17 passed. Without the fix, the 5 new tests (library plus each of the 4 copies) fail.black --line-length=100(24.4.0, as pinned in pre-commit) leaves the new test code and the four copies unchanged.User impact
Anyone running
python -m funasr.metrics.weror the Aishellutils/compute_wer.pyrecipe step. If decoding silently drops utterances, the report now says so instead of claiming every reference was scored.Notes for reviewers
The %WER and %SER numbers are unchanged; only the final summary line's counts change. Missing hypotheses are still skipped rather than scored as deletions, as before.