[oBusk]

diffLines with ignoreWhitespace called trim() on both lines every time equals() was called, and the diff loop calls equals() many times per token. Trimming each line once up front made large diffs with ignoreWhitespace about twice as fast.

The other diffs had the same pattern: ignoreCase, ignoreNewlineAtEof, diffJson's trailing-comma regex and diffWords' trim all ran inside equals(). So this PR moves all of them into a new normalizeToken() hook that runs once per token. equals() now compares the normalized tokens. The returned change objects still contain the original text, and diff output is unchanged (fuzz-tested against master across all diff functions and option combinations).

The README documents normalizeToken() alongside the other methods a Diff subclass can override.

Benchmarks

Median ms, master vs this PR:

Case	master	PR	Speedup
diffLines (no options)	11.1	11.1	1.00x
diffLines ignoreWhitespace	21.8	11.6	1.87x
diffLines ignoreNewlineAtEof	18.7	11.9	1.57x
diffLines ignoreCase	30.8	11.9	2.59x
diffJson	130.8	18.3	7.17x
diffWords	105.8	52.3	2.02x
diffWords ignoreCase	139.3	51.6	2.70x
diffChars (no options)	154.5	130.2	1.19x
diffChars ignoreCase	188.4	130.5	1.44x
diffSentences (no options)	0.78	0.75	1.04x
diffArrays (no options)	16.0	15.3	1.04x

Cases where nothing needs normalizing (diffLines, diffSentences and diffArrays without options) are not slower.

Inputs, all from this repo's history:

• lines and arrays: yarn.lock before and after ff14775 (8.5k lines, ~800 changed)
• diffJson: the same two files parsed into objects
• words and sentences: README.md at 9e2da01 and 8d706c8
• chars: the first 3,000 characters of release-notes.md at the same two commits

Node 26.10.0, 5 alternating runs per build.

Behaviour changes

A comparator now receives normalized tokens

With ignoreCase, it gets lowercased tokens. Previously ignoreCase was silently skipped whenever a comparator was passed. comparator is only documented and typed for diffArrays, which has no ignoreCase option, and I found nothing in the docs, tests, issues or PRs about combining the two. I'm assuming the old behaviour was a side effect of how equals() was written, not a design decision. (diffLines and diffJson already passed their trimmed or comma-stripped tokens to a comparator; only the lowercasing is new.)

diffWords now honours a comparator

Previously its equals() never called it. comparator isn't a documented diffWords option either.

equals() receives normalized tokens

A subclass that overrides equals() still works, but now gets, for example, lowercased tokens when ignoreCase is set. Calling equals() directly on the exported instances (lineDiff.equals(...) etc.) no longer applies options like ignoreWhitespace, because normalizeToken() applies them now. I'm assuming direct calls are rare, since they aren't documented.

diffArrayswithignoreCase` on non-string items no longer throws

when the arrays differ. ignoreCase now leaves non-string tokens alone.

[ExplodingCabbage]

Haven't reviewed the code yet but will comment to note that in concept this sounds good, and also that it happily helps enable a (perhaps more dramatic) further optimization (see #736).

Only thing I need to think about is whether increased memory use is a strike against this, and how to characterise the memory impact in the release notes. (I guess that potentially this roughly doubles peak memory use, in the case where the tokens themselves are large strings and therefore dominate the memory use? My thinking is that previously we'd only have needed to hold on to normalized versions of two of those strings at a time before we exit .equals and they could be deallocated; now I guess we will hold on to them all for as long as the algorithm runs.)

[oBusk]

Good question, I measured it. In short: you're right about the mechanism, but in practice it only matters for options that actually rewrite tokens (mainly ignoreCase), and only when the tokens are large. Peak memory doesn't double even then.

Where the extra memory comes from

• With no normalizing options, normalizeToken() returns the token itself, so the only cost is two arrays of pointers.
• trim() and slice() (ignoreWhitespace, ignoreNewlineAtEof, diffWords) don't copy the text: V8 and JavaScriptCore keep a small reference into the original string instead. When there's nothing to trim, trim() returns the same string.
• toLowerCase() and diffJson's comma replace() do make a full new string for every token they change, and those copies now live for the whole diff. That's the case you described.
• V8's toLowerCase() also copied tokens that were already lowercase. I've pushed 835507c, which keeps the original token when lowercasing doesn't change it. That had no measurable effect on speed.

Peak RSS growth during one diff, in MB (median of 3 runs, Node 26.10 / Bun 1.4.4, with 835507c)

Case	Node, master	Node, PR	Bun, master	Bun, PR
diffLines, yarn.lock (8.5k lines)	16.5	16.5	21.2	21.0
… with ignoreWhitespace	16.3	17.3	21.2	21.6
… with ignoreCase	16.7	16.9	24.9	22.6
diffWords ignoreCase, README	24.5	24.4	24.4	27.4
diffJson	24.5	24.6	24.0	24.2
diffLines ignoreCase, every line ×50 (2 × 18 MB)	83	106	116	109
… same, but the input is all uppercase	116	152	140	129
diffLines ignoreWhitespace, every line ×50	107	116	88	90

At realistic sizes the difference is within noise, apart from about +3 MB for diffWords ignoreCase on Bun. The worst case I could build is huge tokens that all change: on Node that's about +30% peak, roughly one extra copy of the token text. On Bun the PR is often below master, presumably because master created two new strings on every comparison.

Things I tried that didn't help

• Interning (sharing one copy of each distinct normalized token through a Map): helped on some Node cases, made ignoreWhitespace worse, cost 10–15% speed on Node, and did nothing on Bun.
• Avoiding the copy in replace() by slicing off the comma and joining the pieces back together: the engines build the full copy anyway the first time the strings are compared.

The only way to go further would be to stop keeping normalized strings during the diff at all, for example by hashing each normalized token and only re-normalizing to confirm a match when two hashes are equal. That's a bigger change, needs a fallback path for comparator and overridden equals(), and only helps in the worst case above, so I'd leave it out of this PR.

Possible release-notes wording

Tokens are now normalized once up front rather than on every comparison, which makes diffs with options like ignoreCase and ignoreWhitespace, as well as diffJson and diffWords, considerably faster. Options that rewrite tokens, mainly ignoreCase, now keep a normalized copy of each token they change for the length of the diff, resulting in a larger memory usage than previous versions. In the worst case (very large tokens that all change in normalization) this adds up to roughly one extra copy of the input text to peak memory; otherwise memory use is essentially unchanged.

[ExplodingCabbage]

I don't think this is true? My Claude disagrees, and if I test with %DebugPrint I see the result of x.toLowerCase() having the same memory address as x:

mark@fractal:~$ node --allow-natives-syntax
Welcome to Node.js v22.14.0.
Type ".help" for more information.
> x = 'hello world'
'hello world'
> y = x.toLowerCase()
'hello world'
> %DebugPrint(x)
DebugPrint: 0x3a4243dea151: [String] in OldSpace: #hello world
0x3999b7cc0691: [Map] in ReadOnlySpace
 - map: 0x3999b7cc0841 <MetaMap (0x3999b7cc0099 <null>)>
 - type: INTERNALIZED_ONE_BYTE_STRING_TYPE
 - instance size: variable
 - elements kind: HOLEY_ELEMENTS
 - enum length: invalid
 - stable_map
 - non-extensible
 - back pointer: 0x3999b7cc0069 <undefined>
 - prototype_validity cell: 0
 - instance descriptors (own) #0: 0x3999b7cc0c91 <DescriptorArray[0]>
 - prototype: 0x3999b7cc0099 <null>
 - constructor: 0x3999b7cc0099 <null>
 - dependent code: 0x3999b7cc0c51 <Other heap object (WEAK_ARRAY_LIST_TYPE)>
 - construction counter: 0

'hello world'
> %DebugPrint(y)
DebugPrint: 0x3a4243dea151: [String] in OldSpace: #hello world
0x3999b7cc0691: [Map] in ReadOnlySpace
 - map: 0x3999b7cc0841 <MetaMap (0x3999b7cc0099 <null>)>
 - type: INTERNALIZED_ONE_BYTE_STRING_TYPE
 - instance size: variable
 - elements kind: HOLEY_ELEMENTS
 - enum length: invalid
 - stable_map
 - non-extensible
 - back pointer: 0x3999b7cc0069 <undefined>
 - prototype_validity cell: 0
 - instance descriptors (own) #0: 0x3999b7cc0c91 <DescriptorArray[0]>
 - prototype: 0x3999b7cc0099 <null>
 - constructor: 0x3999b7cc0099 <null>
 - dependent code: 0x3999b7cc0c51 <Other heap object (WEAK_ARRAY_LIST_TYPE)>
 - construction counter: 0

'hello world'

Am I missing something? If not, this optimisation definitely seems misguided!

[oBusk]

TL;DR: On current V8 (Node 24+), toLowerCase() copies strings even when nothing changes once they're longer than 24 characters (and non-ASCII strings get copied on every Node version), so the fix saves real memory there, while on other runtimes it's a cheap no-op.

Okay, so that was a rabbit hole! I had benchmarked the fix as actually using less memory, so your test confused me, and when I reran it, Node 26 showed no copy either. So why did the fix reduce memory usage?

I ended up looking at the V8 source code (with Claude's help). toLowerCase() has a fast path, used for strings of at most 24 characters (such as "hello world") with no characters above U+00FF, and it still returns the original if nothing changed. Anything longer goes down another code path, which always copies. That includes slices and joined strings from 13 characters up, which is exactly what diffLines tokens are. Strings with characters above U+00FF (€, 中, emoji) go to ICU, which always copies, at any length.

So I reran your test with a 25+ character string and got the copying behaviour that warrants the fix. But before responding I reran it on Node 22, and there it didn't copy!

That led me to V8 commit d8fd49d (March 2025). It improves toUpperCase() by checking whether there's anything to convert before creating the copy, and so removes the "return the original after copying" check from the shared helper. But it doesn't add the same improvement for toLowerCase(), which lost its check too, and the review even mentions this:

I removed change detection support from the FastConvert case as well now since we really shouldn't convert at all in that case. I didn't add it to ToLower yet though, since it's a bit messy due to direct calls from CSA...

This shipped in V8 13.6, i.e. Node 24. The old check only ever covered pure ASCII, though, so non-ASCII strings past the fast path were already copied on Node 22. Here's x.toLowerCase() on an already-lowercase x, checked with %DebugPrint:

x	Node 22	Node 24+
≤ 24 chars, no characters above U+00FF	original	original
> 24 chars (or a diffLines token ≥ 13 chars), ASCII	original	copy
same, but with Latin-1 (é)	copy	copy
any character above U+00FF, any length	copy	copy

Extra heap for the normalized tokens of diffLines(…, { ignoreCase: true }) on yarn.lock, without → with the fix:

Input	Node 22	Node 24+
yarn.lock (8.5k lines)	194 KB → 196 KB	432 KB → 192 KB (−56%)
every line repeated 50× (17 MB)	5.1 MB → 5.1 MB	17.7 MB → 5.1 MB (−71%)

Current V8 main still copies. I couldn't find any reported issue on this, so I'm thinking of reporting it, to be honest, though I've never reported an issue to Chromium before.

So I'd argue the fix (comparing ourselves and returning the original where appropriate) is useful on every V8-based runtime until this is fixed, and for non-ASCII text even on older ones, and it shouldn't hurt anything else: on Bun it's a no-op, and I couldn't measure any difference in speed.

[oBusk]

I had another look at interning the values as we normalize them, and claude drafted (I haven't cleaned it up at all) a branch which interns all values using a hash table. Interning keeps only one copy of each distinct normalized string, so with ignoreCase on all-caps input, peak memory end-to-end is lower (143.6 → 108.0 MB on Node, long lines). In the other cases it's about the same or slightly higher (e.g. 113.8 → 122.8 MB with ignoreWhitespace).

One thing to note is that interning is not valid when a consumer is using .comparator or their own overriden .equals() , as those functions expects the token, and not only an ID, so the branch falls back to passing the normalized tokens in those cases.

One unexpected finding is that #734 actually already reduces peak memory, and in most scenarios, it reduces more than interning does. In my first version, which used string Map keys, stacking (new tokenizer + interning) undid much of the memory reduction of the updated tokenizer on Node, because V8 copies strings used as Map keys. The linked branch avoids most of that with its own hash table, but hashing every character in JS makes long-line diffs 2.6–3.8x slower on Node.

I think this PR can be merged as is (no interning, optimize memory where appropiate), and #734 could be reviewed and merged, and then interning/hashing could be investigated later, or perhaps as a precursor to the token discarding PR discussed in #736