Bump max-model-len headroom from +200 to +256 (block-size alignment)#1104
Bump max-model-len headroom from +200 to +256 (block-size alignment)#1104Ankur-singh wants to merge 2 commits into
Conversation
Some attention backends require the KV-cache block count to be a multiple of 128 tokens. With the default block_size=32 that means block_num must be divisible by 4 (128/32). A padding of +200 on top of ISL+OSL can land outside that constraint and the server fails with: RuntimeError: Worker failed with error 'Expected block_num % (128 / block_size) == 0, got block_num=295 and block_size=32' +256 (and any multiple of 128) keeps the context length aligned. This change is propagated to setup_eval_context as well so eval runs inherit the same aligned context.
|
Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook If it is not, please create a PR first before we can merge your PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow If additional help is needed, PR authors can reach out to core maintainers over Slack. |
Some attention backends require the KV-cache block count to be a multiple of 128 tokens. With block_size=32 that means block_num must be divisible by 4. The previous ISL + OSL + 200 padding did not guarantee this alignment (e.g. ISL=8192, OSL=1024 rounds to 295 blocks, not divisible by 4), triggering: RuntimeError: Expected block_num % (128 / block_size) == 0, got block_num=295 and block_size=32 Bumping the headroom to +256 (any multiple of 128) keeps the context length aligned. Mirrors the fix from #1104. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Some attention backends require the KV-cache block count to be a multiple of 128 tokens. With block_size=32 that means block_num must be divisible by 4. The previous ISL + OSL + 200 padding did not guarantee this alignment (e.g. ISL=8192, OSL=1024 rounds to 295 blocks, not divisible by 4), triggering: RuntimeError: Expected block_num % (128 / block_size) == 0, got block_num=295 and block_size=32 Bumping the headroom to +256 (any multiple of 128) keeps the context length aligned. Mirrors the fix from #1104. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| if [ "$ISL" = "1024" ] && [ "$OSL" = "1024" ]; then | ||
| CALCULATED_MAX_MODEL_LEN=$((ISL + OSL + 20)) |
There was a problem hiding this comment.
🔴 The 1k/1k branch in gptoss_fp4_b200.sh still uses ISL+OSL+20 (2068 tokens → 65 blocks → 65%4=1≠0), producing the same 'Expected block_num % (128 / block_size) == 0' RuntimeError the PR was written to fix. Change +20 to +128 or +256 (e.g. CALCULATED_MAX_MODEL_LEN=$((ISL + OSL + 256))) to restore alignment for this branch too.
Extended reasoning...
What the bug is and how it manifests
The PR correctly identifies that attention backends using FlashInfer require block_num % (128 / block_size) == 0. With block_size=32 that means block count must be divisible by 4. The PR fixed the elif branch (8k/1k) by bumping headroom from +200 to +256, but left the if branch (1k/1k) at the original +20 headroom. Any run with ISL=1024 and OSL=1024 on the B200 script will still crash with the identical RuntimeError post-merge.
The specific code path that triggers it
benchmarks/single_node/gptoss_fp4_b200.sh, lines 24-25:
if [ "$ISL" = "1024" ] && [ "$OSL" = "1024" ]; then
CALCULATED_MAX_MODEL_LEN=$((ISL + OSL + 20))This sets max-model-len = 2068. The server is then launched with VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8=1 (line 48), activating the FlashInfer MOE MXFP4 backend for all runs on this script, including the 1k/1k case.
Why existing code does not prevent it
The VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8=1 export is unconditional — it applies regardless of which ISL/OSL branch calculated CALCULATED_MAX_MODEL_LEN. The 8k/1k fix was applied only to the elif branch; the if branch was overlooked.
What the impact would be
Running the B200 benchmark with ISL=1024, OSL=1024 post-merge will crash immediately at server startup with: RuntimeError: Worker failed with error 'Expected block_num % (128 / block_size) == 0, got block_num=65 and block_size=32'. The 1k/1k configuration is a standard benchmark point, so this crash will be hit in normal CI/benchmark runs.
How to fix it
Change line 25 from $((ISL + OSL + 20)) to any value whose total is divisible by 128 (so the block count is divisible by 4). Using +256 (consistent with the elif branch fix) gives 2048+256=2304, 2304/32=72, 72%4=0. Using +128 also works: 2048+128=2176, 2176/32=68, 68%4=0. The same fix should be applied to gptoss_fp4_h200.sh for consistency (though the H200 script uses VLLM_MXFP4_USE_MARLIN=1 instead of FlashInfer, so the alignment constraint may not be strictly required there).
Step-by-step proof
ISL=1024,OSL=1024hits theifbranch:CALCULATED_MAX_MODEL_LEN = 1024+1024+20 = 2068- vLLM server starts with
max-model-len=2068and defaultblock_size=32 - Block count =
ceil(2068/32) = ceil(64.625) = 65blocks - Alignment check:
65 % (128/32) = 65 % 4 = 1 ≠ 0— FAILS - Error:
RuntimeError: Expected block_num % (128 / block_size) == 0, got block_num=65 and block_size=32 - Compare to the fixed elif branch:
8192+1024+256=9472,9472/32=296,296%4=0— passes
* Add B300 config: kimi-k2.5-fp4-vllm - Add kimik2.5-fp4-b300-vllm benchmark config and the corresponding benchmarks/single_node/kimik2.5_fp4_b300.sh launch script - At the time of submission, the vLLM Kimi-K2.5 recipes page does not have a B300-specific recipe, so this reuses the existing Kimi-K2.5 FP4 (NVFP4) B200 vLLM recipe as-is until B300-specific tuning is available - Image: vllm/vllm-openai:v0.17.0 (same as B200), runner: b300, same TP/EP/concurrency search-space as B200 * Bump max-model-len headroom from +200 to +256 (block-size alignment) Some attention backends require the KV-cache block count to be a multiple of 128 tokens. With block_size=32 that means block_num must be divisible by 4. The previous ISL + OSL + 200 padding did not guarantee this alignment (e.g. ISL=8192, OSL=1024 rounds to 295 blocks, not divisible by 4), triggering: RuntimeError: Expected block_num % (128 / block_size) == 0, got block_num=295 and block_size=32 Bumping the headroom to +256 (any multiple of 128) keeps the context length aligned. Mirrors the fix from #1104. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: functionstackx <47992694+functionstackx@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Looks like the changes are already in |
|
@Ankur-singh not for b200, can u trigger an re-run on b200 i only did it for b300 |
|
You mean all b200 configs or just Kimi? |
2 participants
Summary
Some attention backends require the KV-cache block count to be a multiple of 128 tokens. With the default
block_size=32that meansblock_nummust be divisible by128 / 32 = 4. The currentISL + OSL + 200padding doesn't guarantee this alignment — for example atISL=8192,OSL=1024the context length rounds to 295 blocks which is not divisible by 4:```
RuntimeError: Worker failed with error 'Expected block_num % (128 / block_size) == 0, got block_num=295 and block_size=32', please check the stack trace above for the root cause
```
Bumping the headroom to +256 (any multiple of 128 — even
+128would suffice) keeps the context length aligned and avoids this crash. Also propagated tosetup_eval_contextso eval runs use the same aligned context.Changes
Test plan