compute_wer_by_line raises OverflowError at 32,768 tokens with NumPy 2.3.5 #3781

🐛 Bug

compute_wer_by_line uses an np.int16 edit-distance matrix. When either input reaches 32,768 tokens, initializing the first row or column raises OverflowError: Python integer 32768 out of bounds for int16 under NumPy 2.3.5.

This can be reproduced with one input containing 32,768 tokens and the other containing only one token. The matrix has just 65,538 cells; the failure does not require two long inputs or a large quadratic allocation. All tokens can be identical, so the expected edit distance is 32,767, which itself fits in int16. The initialization still tries to store 32,768.

Environment

• Repository: modelscope/FunASR
• Commit: 879a219c717123fdec6024e8e56c734a7aace2ff
• Source: funasr/metrics/wer.py
• Python 3.12.14
• NumPy 2.3.5

To Reproduce

Save the code sample below as repro_wer.py in the checkout root and run python repro_wer.py.

Code sample

From a checkout of the commit above with NumPy 2.3.5 installed, run the following. This loads the exact upstream function via Python's AST so the reproduction does not require unrelated ASR dependencies or importing the full FunASR package.

import ast
from pathlib import Path
import numpy as np

path = Path("funasr/metrics/wer.py")
tree = ast.parse(path.read_text())
function = next(
    node for node in tree.body
    if isinstance(node, ast.FunctionDef)
    and node.name == "compute_wer_by_line"
)
namespace = {"np": np}
exec(compile(ast.Module(body=[function], type_ignores=[]), str(path), "exec"), namespace)
compute_wer_by_line = namespace["compute_wer_by_line"]

for n in (32766, 32767, 32768, 32769):
    for hyp, ref in ((["a"] * n, ["a"]), (["a"], ["a"] * n)):
        try:
            result = compute_wer_by_line(hyp, ref)
            print(len(hyp), len(ref), "wrong:", result["wrong"])
        except OverflowError as exc:
            print(len(hyp), len(ref), type(exc).__name__ + ":", exc)

Error logs

Observed output:

32766 1 wrong: 32765
1 32766 wrong: 32765
32767 1 wrong: 32766
1 32767 wrong: 32766
32768 1 OverflowError: Python integer 32768 out of bounds for int16
1 32768 OverflowError: Python integer 32768 out of bounds for int16
32769 1 OverflowError: Python integer 32768 out of bounds for int16
1 32769 OverflowError: Python integer 32768 out of bounds for int16

The exception occurs at cost_matrix[i][0] = i (line 142) when the hypothesis is long, or cost_matrix[0][j] = j (line 144) when the reference is long, before the main dynamic-programming loop.

Expected behavior

For the 32,768-vs-1 identical-token case, return an edit distance of 32,767 with one correct token and 32,767 insertions (long hypothesis) or deletions (long reference), instead of failing during initialization.

The cost matrix should use an integer representation that can safely hold the initialized lengths and edit costs for supported inputs.

Scope of verification

This report verifies the function from the pinned commit above on Python 3.12.14 / NumPy 2.3.5. Both input directions were tested at 32,766, 32,767, 32,768 and 32,769 tokens, with the other input fixed at one token. Passing controls were also checked for correct, insertion, deletion and substitution counts. Earlier NumPy versions, full ASR inference and the CLI were not tested. No claim is made here about behavior on other NumPy versions or later FunASR revisions.

Audio details

Not applicable: synthetic token lists only, no model, audio, GPU, PyTorch or torchaudio used. Source function loaded without importing the full package.

Would a focused change to the edit-distance matrix integer representation be welcome for this boundary? I am reporting the reproduction before making changes.