[AMD] Update MiniMax-M3-MXFP4 MI355X vLLM disagg perf and config / 更新 MiniMax-M3-MXFP4 MI355X vLLM disagg 性能与配置

[Duyi-Wang]

Changes

• Prefill-only INT4 quick-reduce: set VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 and VLLM_ROCM_QUICK_REDUCE_MAX_SIZE_BYTES_MB=2048 on the prefill workers, via a new prefill_env channel in server_vllm.sh (mirrors the existing decode_env path; injected on every prefill rank). Applied to the MiniMax-M3-MXFP4 entry in models_vllm.yaml.
• Search space: cap the 1P1D TP4 concurrency sweep at 256 (was 512). Drop 2P1D TP4.

Notes

• Quick-reduce is prefill-only (the all-reduce path); decode is unaffected.
• Single commit on top of main; perf-changelog.yaml entry included.

中文说明

为 MiniMax-M3-MXFP4 MI355X vLLM disagg 配置新增 prefill 端 INT4 quick-reduce 优化（设置 VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 和 VLLM_ROCM_QUICK_REDUCE_MAX_SIZE_BYTES_MB=2048），通过 server_vllm.sh 中新增的 prefill_env 通道注入到每个 prefill rank。调整搜索空间：将 1P1D TP4 并发上限从 512 降至 256，丢弃2P1D TP4。

[github-actions]

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

• https://docs.sglang.io/cookbook/intro
• https://recipes.vllm.ai/

If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

• https://github.com/vllm-project/recipes
• https://github.com/sgl-project/sglang/tree/main/docs_new

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.

If additional help is needed, PR authors can reach out to core maintainers over Slack.

---

感谢你的贡献！对于 vLLM 与 SGLang，请确保你的 recipe 与官方 vLLM recipes 和/或 SGLang cookbook 保持一致

• https://docs.sglang.io/cookbook/intro
• https://recipes.vllm.ai/

如果不一致，请先创建一个 PR，之后我们才能将你的单节点 PR 合并到 master 分支。让我们确保文档保持一流水准，使整个 ML 社区都能从你的辛勤工作中受益！谢谢

• https://github.com/vllm-project/recipes
• https://github.com/sgl-project/sglang/tree/main/docs_new

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。如果选择重新运行失败的任务，PR 作者有责任确保其最终通过。参见 GitHub 关于重新运行失败任务的文档：https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

一般而言，PR 作者应先向相应公司的 CODEOWNERS 请求审阅并获得 PR 批准，然后再请求核心维护者审阅。

如需更多帮助，PR 作者可通过 Slack 联系核心维护者。

[claude]

Claude finished @Duyi-Wang's task in 1m 3s —— View job

---

PR Review

• Gather context (diff, server_vllm.sh, models_vllm.yaml, perf-changelog.yaml)
• Verify prefill_env injection mirrors decode_env
• Validate master-config + perf-changelog rules

LGTM — no blocking issues found.

Verified:

• The new prefill_env / PREFILL_MODEL_ENVS channel correctly mirrors the existing decode_env path. It's injected on all prefill ranks — node 0 (proxy+prefill, server_vllm.sh:256) and additional prefill nodes 1..xP-1 (server_vllm.sh:430) — before each vllm serve prefill command. The for env_pair in ${PREFILL_MODEL_ENVS}; do export "$env_pair" pattern word-splits the two KEY=VALUE pairs correctly. Decode is untouched, matching the "prefill-only" intent.
• amd-master.yaml was modified and perf-changelog.yaml was updated — the new entry is correctly appended to the end of the file (after PR [codex] add MiniMax M3 FP4 MI355X vLLM MTP benchmark #1939's entry).
• The minimaxm3-fp4-mi355x-vllm-disagg config key is pre-existing (only the conc-list was edited 512→256), so no new model-prefix concern introduced here.

[functionstackx]

can u update this an upstream docker image such that it is a performance update PR? @Duyi-Wang

[Duyi-Wang]

Ok, the image will be updated once all optimizations have been merged.

[functionstackx]

okay feel free to reopen an new PR once there is acutally an upstream image such that

1. u dont need have an hard time resolving conflicts
2. we dont waste gpu time on testing images that we not gotta accept

[claude]

Claude finished @Duyi-Wang's task in 1m 4s —— View job

---

Re-review

• Diff new changes since last review (2026-06-26 LGTM)
• Validate upstream image swap + new env var
• Re-check perf-changelog ordering & master-config rules

Re-review: LGTM — no blocking issues found.

What changed since my last review (addresses @functionstackx's request to use an upstream image):

• Image rocm/vllm-dev:vllm-0.23.1-rocm723-mi35x-mori-0625 → vllm/vllm-openai-rocm:nightly-4559c43a9526597c00cbcc4f59979496500268d1. This is now a public Docker Hub org/image:tag reference (upstream), so this is a proper perf-update PR as requested. ✅
• New shared env VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 added to env:. Confirmed it is injected via setup_vllm_env → MODEL_ENVS (server_vllm.sh:229), which runs on all node roles (proxy+prefill, prefill, decode), so it correctly applies to both prefill and decode as the changelog states.
• perf-changelog.yaml entry updated to document the image bump and is still correctly appended to the end of the file (after PR fix: recover PR 1931 ingest via sweep reuse #1966).

Previously-verified items still hold:

• prefill_env / PREFILL_MODEL_ENVS is prefill-only, injected on node 0 (server_vllm.sh:256) and prefill ranks 1..xP-1 (server_vllm.sh:430); decode path (DECODE_MODEL_ENVS, :486) is untouched. Word-splitting of the two KEY=VALUE pairs is correct.
• conc-list 512→256 for 1P1D TP4; config key minimaxm3-fp4-mi355x-vllm-disagg is pre-existing.

Note: I can't verify the specific nightly tag exists in the vllm/vllm-openai-rocm repo — CI's image pull will confirm that.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28489114424
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=28489114424

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28489114424
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=28489114424

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28489114424
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=28489114424

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28646019011
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=28646019011

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28764297239
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=28764297239

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28764297239
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=28764297239

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28853536583
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=28853536583

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28853536583
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=28853536583

[billishyahao]

The latest sweep and eval link: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/28853536583

[billishyahao]

/reuse-sweep-run 28853536583

[billishyahao]

As a PR reviewer and CODEOWNER, I have reviewed this and have:

• Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
• Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
• Verified that this PR has passed PR validation. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/28853536583.
• Verified that this PR passes evals. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/28853536583.
• Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
• Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
• If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
• If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
• Verified that the single-node recipes are similar to the official vLLM recipes and/or theSGLang cookbook:
  • If they are not, I have verified that a PR has been opened in vLLM recipe repo or SGLang repo and linked it below in the additional detail section:
• If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.

Additional detail section:

• insert any additional info here

Signed: billishyahao

[Klaud-Cold]

✅✅✅ Verdict: PASS ✅✅✅

✅ Check 0 (CODEOWNER): PASS — billishyahao is a named owner of configs/amd-master.yaml; remaining files are catch-all-only.
✅ Check 1 (sweep on in-PR commit): PASS — commit bc80e2bd (in PR) has green executed multi-node 8k1k / and multi-node eval / check-runs in run 28853536583; single-node lanes skipped only because this PR has no single-node configs; head commit is a no-op merge of main.
✅ Check 2 (evals pass): PASS — GSM8K em_strict 0.942 (n_eff 1319) on the exact PR config, same image vllm/vllm-openai-rocm:nightly-2dfaae752b... as the master config.
➖ Check 3 (recipe link): N/A — disaggregated/multi-node submission (multinode: true, disagg: true); the recipe-link requirement applies to single-node recipes only.
✅ Check 4 (reuse command): PASS — /reuse-sweep-run 28853536583 posted by billishyahao (COLLABORATOR).
✅ Check 5 (latest checklist): PASS — all current-template items present and checked; the conditional final item is legitimately unchecked.
✅ Check 6 (upstream image / engine-first): PASS — PR moves the image from vendor rocm/vllm-dev to upstream vllm/vllm-openai-rocm on MI355X; framework is vLLM so engine-first ordering does not apply.
✅ Check 7 (no architecture hacks): PASS — changes are AITER shared-expert fusion and prefill-only INT4 quick-reduce (quantized all-reduce, lower-precision communication); no --hf-overrides or FLOPs removal, evals pass.
➖ Check 8 (spec-decode chat template): N/A — no speculative-decoding changes (spec-decoding: "none").

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28916955742
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=28916955742