[cknitt]

Brings back benchmark alerts in CI, this time on a number that doesn't depend on the runner: the words the compiler allocates. The syntax benchmark job from #7100 alerted on wall-clock times above 105% and was disabled (#7380), then removed (#9147), because shared runners vary by more than that.

How it works

A new alloc-benchmarks job runs on pull requests. It builds the compiler, runtime and Belt for the PR and for the base branch (the merge commit's first parent, in a git worktree), and measures both with scripts/allocBenchmarks.js. It fails when a benchmark allocates more than 1% more.

The benchmarks:

• Runtime and Belt, file by file: each of the ~240 sources is parsed and compiled again with the arguments rescript compiler-args gives for it. That covers the parser, type checker, Lambda passes and JS generation on real code.
• Stress inputs: generated files for parts that were superlinear before: a 1500-case int switch, a 400-tag polymorphic variant switch, 2000 let rec bindings, a 300-field record with spreads, a 3000-element list literal, and a 300-function inlining chain.
• Parse and format: bsc -bs-ast and bsc -format on four large files (up to 645 KB), since a build never runs the formatter.

Counts come from OCAMLRUNPARAM=v=0x400 (allocated_words at exit), so no instrumentation library or code change is needed. The stress inputs and the parse/format files come from the PR's checkout for both sides, so their inputs are identical.

If the PR changes any runtime or Belt source, the runtime and Belt rows compile different code on the two sides. They're then listed as "(inputs changed)" with a note, but don't count as regressions. Each file is compiled against the others' interfaces, so one changed source marks all of them.

Syntax benchmarks removed

The allocation comparison makes tests/syntax_benchmarks and make bench redundant, so they're removed. Their four data files move to tests/alloc_benchmarks/data. The tool also undercounted allocations on OCaml 5: Gc.quick_stat minor words only count up to the last minor collection, so Parse RedBlackTree.res reported 0 words. The branch's first commit fixed that before the tool was removed; with the squash merge only the removal remains.

Reporting

The report goes to the job summary and to one comment on the pull request, posted on every run and edited in place on each push (comment-tag: alloc-benchmarks):

• Nothing changed: a single line, "✅ None of the 253 benchmarks changed by 0.1% or more." Changes below 0.1% are the path-length difference between the two checkouts (about 30 words per file) and are never shown.
• Otherwise: a table of the changes of 0.1% or more, in either direction. Growths and savings beyond the threshold get a headline, e.g. "⚠️ Regression: 1 benchmark allocates more than 1% more."
• Footer: names the PR and base commits and links the run.
• Status symbol: in the title and on the headlines: ⚠️ for a regression, 🎉 for an improvement without regressions, ✅ otherwise (also when only "(inputs changed)" rows appear).

Fork PRs only get the job summary, because their token can't comment.

The job is not in ci-ok, so a deliberate allocation increase doesn't block a merge. The red check and the comment are the alert.

Measured

Locally (linux-arm64 devcontainer):

• Same binary twice: all counts identical.
• Same compiler in two checkouts: every difference below 0.1%, from path lengths only.
• An injected regression (one 100-word array per type_expect call): total +2.54%, 55 benchmarks over 1%, job exits 1. Largest changes: stress/ListLiteral.res +16%, stress/LetRec.res +14%, Belt_Array.res +6.8%.
• Peak heap (top_heap_words) is not usable. It moved by up to ±6.7% between the two checkouts above, because heap growth depends on when collections happen, so it isn't reported.

In CI the job took 1 min 44 s:

• building the base: 44 s, mostly from the dune cache shared with the linux-arm64 build job;
• building head: 3 s, reusing master's cached build;
• measuring: about 9 s per side.

It runs in parallel with build-compiler, so it doesn't lengthen CI.

New tests

scripts/allocBenchmarks.test.js (yarn test-alloc-benchmarks, run at the start of the job) covers:

• parsing the runtime's statistics;
• the threshold, for regressions and improvements;
• "(inputs changed)" rows not counting as regressions, and their note;
• added and removed benchmarks;
• the 0.1% table cut-off and the no-change line;
• singular and plural headlines;
• the stress generators producing the same input every time.

Later: instruction counts

Allocation misses slowdowns that don't allocate, such as scans over existing lists. Cachegrind instruction counts could cover those. The estimate is about 1 min more for the stress inputs and parse/format files, and 2–3 min if the ~15 largest runtime/Belt files are added. Running all ~500 compiles under valgrind would take 10–15 min. bsc crashed under valgrind on arm64, so this part would run on an x86 runner.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Kbf3gG86E3DicyC4NfsFGP

[github-actions]

Changelog

🏠 Internal

• Compare the words the compiler allocates with the base branch in CI, and fail on a growth of more than 1%.
• Remove the syntax benchmarks (make bench); the CI comparison parses and formats their files with bsc instead.

Semantic Diff

1 commit · Full page

✏️ Compare the compiler's allocations with the base branch in CI

_{4e9aa3e · cknitt + Claude · Nov 07, 2026, 07:17 UTC · edited}

Timings on shared runners are too noisy to alert on, which is why the previous benchmark job was removed. The words the compiler allocates are the same in every run, so a new job measures them for the base branch and the pull request, fails when a benchmark allocates more than 1% more, and comments the comparison on the pull request.

The syntax benchmarks and make bench are removed; the comparison parses and formats their files with bsc, from tests/alloc_benchmarks/data.

[github-actions]

✅ Allocation benchmarks

Allocated words in total: 384,473,552 → 384,496,081 (+0.01%).

✅ None of the 253 benchmarks changed by 0.1% or more.

Pull request 4e9aa3e compared with base 827be16 in this run.

[cknitt]

@codex review

[chatgpt-codex-connector]

Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-07T07:22:29.127217Z	4e9aa3e	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: 4e9aa3e057

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".