Skip to content

fix(metrics): size the WER cost matrix for long inputs - #3784

Merged
LauraGPT merged 1 commit into
modelscope:mainfrom
Batianxieshen1:fix-wer-int16-overflow
Oct 20, 2026
Merged

LauraGPT merged 1 commit into
modelscope:mainfrom
Batianxieshen1:fix-wer-int16-overflow

Conversation

@Batianxieshen1

Copy link
Copy Markdown
Contributor

Summary

compute_wer_by_line allocated its edit-distance matrix as int16. An edit
distance can be as large as max(len_hyp, len_ref), so initialising the first
row/column raises OverflowError as soon as either input reaches 32768 tokens:

OverflowError: Python integer 32768 out of bounds for int16

This reproduces with one 32768-token hypothesis and a one-token reference — the
matrix is only 65538 cells, so it does not take two long inputs to hit. All
tokens can be identical, and the resulting distance (32767) still fits in int16;
it is the initialisation storing the index 32768 that overflows.

Fix: use int32, which covers any input this O(len_hyp * len_ref) loop can
realistically reach. Results for existing inputs are unchanged.

Fixes #3781

Type of change

  • Bug fix
  • Documentation
  • Example or demo
  • Runtime or deployment
  • Benchmark or evaluation
  • Model/training change

Validation

  • python -m compileall funasr examples tests (exit 0; only pre-existing
    SyntaxWarnings in examples/industrial_data_pretraining/fun_asr_nano/tools/format5res.py)
  • Docs or links checked — not applicable
  • Runtime/deployment command tested — not applicable (pure function)

Manual verification on NumPy 2.3.5:

  • Before: compute_wer_by_line(["a"] * 32768, ["a"]) raised
    OverflowError: Python integer 32768 out of bounds for int16.
  • After: returns {'nwords': 1, 'cor': 1, 'wrong': 32767, 'ins': 32767, 'del': 0, 'sub': 0}.
  • The mirrored case compute_wer_by_line(["a"], ["a"] * 32768) returns
    {'nwords': 32768, 'cor': 1, 'wrong': 32767, 'ins': 0, 'del': 32767, 'sub': 0}.
  • 3000 random small inputs (0–25 tokens each side, alphabet of 3) produce
    identical compute_wer_by_line results before and after the change, so
    normal-scoring behaviour is untouched.

Two regression tests were added to tests/test_metrics_wer.py covering both
directions of the boundary:

python -m pytest tests/test_metrics_wer.py -q
4 passed in 5.96s

User impact

Anyone scoring long transcripts or concatenated utterances with
compute_wer_by_line (including compute_wer and the CER detail report) no
longer crashes once a line reaches 32768 tokens; such inputs now produce the
correct edit distance instead of an exception.

Note

The same dtype=np.int16 pattern exists in four standalone copies of this
helper — runtime/python/utils/compute_wer.py:99,
runtime/python/libtorch/funasr_torch/utils/compute_wer.py:99,
examples/aishell/paraformer/utils/compute_wer.py:103 and
examples/aishell/e_paraformer/utils/compute_wer.py:103. I kept this PR limited
to the library function and its tests; happy to send a follow-up for those if
you want them aligned.

compute_wer_by_line allocated the edit-distance matrix as int16. An edit
distance can be as large as max(len_hyp, len_ref), so the first row/column
initialisation raises OverflowError as soon as either input reaches 32768
tokens, e.g. one 32768-token hypothesis against a one-token reference.

Use int32, which covers any input this O(len_hyp * len_ref) loop can
realistically reach. Results for existing inputs are unchanged.

Fixes modelscope#3781

Signed-off-by: Batianxieshen1 <280706568+Batianxieshen1@users.noreply.github.com>

@LauraGPT LauraGPT left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Validated against current main 879a219c717123fdec6024e8e56c734a7aace2ff and exact head abc71f8019882a1ff598493858b60ed0cddadb34.

The submitted boundary tests reproduce the NumPy int16 OverflowError on the base (2 failures / 4 tests) and pass on both the exact head and current synthetic merge (4/4 each). The adjacent RapidFuzz metric tests pass 2/2 on both trees, changed-file compilation passes, and recursive tree reconciliation shows only funasr/metrics/wer.py and tests/test_metrics_wer.py differ from current main. The int32 change is appropriately scoped for this CPU-only edit-distance helper; no model or GPU runtime behavior is changed.

@LauraGPT
LauraGPT merged commit 1d066f9 into modelscope:main Oct 20, 2026
LauraGPT pushed a commit that referenced this pull request Oct 21, 2026
…3785)

Four standalone copies of compute_wer_by_line still allocated the edit-distance
matrix as int16: the runtime bundles (runtime/python/utils and
runtime/python/libtorch/funasr_torch) and the two Aishell examples. They raise
the same OverflowError once either input reaches 32768 tokens, which #3784
fixed for the library copy in funasr/metrics/wer.py.

Apply the same int32 sizing, and cover every copy from
tests/test_metrics_wer.py so the duplicates cannot silently regress: one
boundary case per file plus an agreement check against the library helper.

Signed-off-by: Batianxieshen1 <280706568+Batianxieshen1@users.noreply.github.com>
Co-authored-by: Batianxieshen1 <280706568+Batianxieshen1@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

compute_wer_by_line raises OverflowError at 32,768 tokens with NumPy 2.3.5

2 participants