Skip to content

[Klaud Cold] Remove dsv4-fp8-mi355x-sglang recipe#1501

Merged
functionstackx merged 1 commit into
mainfrom
remove-dsv4-fp8-mi355x-sglang
May 18, 2026
Merged

[Klaud Cold] Remove dsv4-fp8-mi355x-sglang recipe#1501
functionstackx merged 1 commit into
mainfrom
remove-dsv4-fp8-mi355x-sglang

Conversation

@functionstackx

Copy link
Copy Markdown
Collaborator

Summary

Removes the dsv4-fp8-mi355x-sglang recipe (and its orphan launch script benchmarks/single_node/dsv4_fp8_mi355x.sh). The current sglang ROCm MI355X image lineage cannot serve the DSV4-Pro-FP8 checkpoint without errors.

Why

Bump PR #1470 tried to advance the image from rocm/sgl-dev:deepseek-v4-mi35xlmsysorg/sglang-rocm:v0.5.12-rocm720-mi35x-20260517. Every sweep job on the new image failed at server boot:

ValueError: Downcasting not allowed: target.dtype=torch.float8_e4m3fn,
loaded_weight.dtype=torch.bfloat16
File ".../sglang/srt/models/deepseek_v4.py:1544"
File ".../sglang/srt/utils/parameter.py:73 copy_with_check"

DSV4-Pro-FP8 has some bf16 layers (shared-expert / embedding); the old rocm/sgl-dev:deepseek-v4-mi35x was a custom build that tolerated this, the generic v0.5.12 ROCm MI355X release added a strict downcast check that we can't satisfy without an upstream change.

Path forward: if we re-introduce dsv4-fp8 on mi355x later, we'll either need (a) a re-bake of the custom image, (b) an upstream sglang change relaxing the downcast check, or (c) a DSV4 checkpoint that's pure fp8 with no bf16 layers. None of those are immediate, so leaving the recipe in place creates a permanently-FAILED config in every cron sweep — better to remove it cleanly.

Diff

  • .github/configs/amd-master.yaml: drop dsv4-fp8-mi355x-sglang block (lines 1709-1727)
  • benchmarks/single_node/dsv4_fp8_mi355x.sh: deleted (no remaining recipe references it)

Other dsv4-mi355x recipes (dsv4-fp4-mi355x-sglang, dsv4-fp4-mi355x-atom, dsv4-fp8-mi355x-vllm, dsv4-fp8-mi355x-atom) are untouched.

Test plan

  • YAML loads (python3 -c "import yaml; yaml.safe_load(open('.github/configs/amd-master.yaml'))").
  • No other config / script / doc references dsv4-fp8-mi355x-sglang or dsv4_fp8_mi355x.sh.
  • Admin-merging without sweep (recipe removal — nothing to benchmark).

🤖 Generated with Claude Code

The v0.5.12-rocm720-mi35x-20260517 image rejects bf16 weights in fp8
slots (ValueError: Downcasting not allowed at deepseek_v4.py:1544 →
parameter.py:73 copy_with_check) — DSV4-Pro-FP8 has bf16 shared-expert/
embedding layers and the old rocm/sgl-dev:deepseek-v4-mi35x image
tolerated this, the generic v0.5.12 ROCm image doesn't.

Bump PR #1470 root-caused via the failing sweep; the recipe isn't
viable on any tag in the current sglang ROCm MI355X lineage until
upstream relaxes the downcast check or ships a converter. Closing
that PR; if the recipe is needed again later it can be re-added with
whichever image works at that point.

Also removes the orphan launch script benchmarks/single_node/
dsv4_fp8_mi355x.sh (not referenced by any remaining recipe).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@functionstackx functionstackx requested a review from a team May 18, 2026 17:31
@functionstackx functionstackx merged commit adbaae5 into main May 18, 2026
@functionstackx functionstackx deleted the remove-dsv4-fp8-mi355x-sglang branch May 18, 2026 17:31
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.

If additional help is needed, PR authors can reach out to core maintainers over Slack.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — clean removal of a permanently-failing recipe and its orphan launch script.

Extended reasoning...

Overview

This PR removes the dsv4-fp8-mi355x-sglang recipe block from .github/configs/amd-master.yaml and deletes its launch script benchmarks/single_node/dsv4_fp8_mi355x.sh. The motivation is sound and clearly explained: the upstream sglang ROCm MI355X image added a strict downcast check that rejects DSV4-Pro-FP8's bf16 layers, and the prior custom image is no longer being maintained, so this recipe was permanently failing in cron sweeps.

Security risks

None. This is a configuration removal — no auth, crypto, permissions, or input handling involved.

Level of scrutiny

Low. Pure deletion of a failing benchmark recipe and its orphan script. The change is mechanical, the blast radius is contained to one cron-sweep configuration, and the only externally-visible effect is one fewer (broken) job in the sweep.

Other factors

I verified the test plan claims: outside the deleted lines, the only remaining dsv4-fp8-mi355x references are in perf-changelog.yaml (historical log entries — appropriate to leave) and a comment referencing dsv4_fp8_mi355x_vllm.sh, which is a separate vllm script that still exists on disk. No orphan references to the deleted artifacts remain. Other dsv4-mi355x recipes (fp4-sglang, fp4-atom, fp8-vllm, fp8-atom) are untouched as advertised. No bugs reported by the bug hunting system.

cquil11 added a commit that referenced this pull request May 27, 2026
…xups

Resolutions:
- perf-changelog.yaml: took main verbatim.
- runners/launch_b300-nv.sh: took main (drops --nodelist pin entirely;
  supersedes our narrower 017-019 fix).
- benchmarks/single_node/fixed_seq_len/dsv4_fp8_mi355x{,_vllm}.sh:
  accepted main's deletes (orphan recipes removed in #1374, #1501).
- .github/configs/amd-master.yaml: took main as the base, then re-applied
  our agentic-only additions on top:
    * qwen3.5-fp8-mi355x-sglang-agentic-hicache  (new entry)
    * dsv4-fp4-mi355x-vllm-agentic               (new entry)
    * dsv4-fp4-mi355x-sglang-agentic             (new entry)
    * kimik2.5-fp4-mi355x-vllm-agentic           (cpu -> lmcache)
  Dropped our comment-path edit for dsv4_fp8_mi355x_vllm.sh since main
  deleted that entry.

Fixed_seq_len reorg fixups for files added on main during our branch's
lifetime:
- git mv 14 stranded scripts from benchmarks/single_node/*.sh into
  benchmarks/single_node/fixed_seq_len/ (dsr1_fp4_b200_mtp,
  dsr1_fp4_mi355x_mtp, dsr1_fp8_h200_mtp, dsr1_fp8_mi325x_mtp,
  dsr1_fp8_mi355x_mtp, dsv4_fp4_mi355x_vllm, glm5_fp8_h200_mtp,
  glm5_fp8_mi325x, glm5_fp8_mi325x_mtp, qwen3.5_bf16_mi325x_mtp,
  qwen3.5_fp4_mi355x_mtp, qwen3.5_fp8_h100, qwen3.5_fp8_h100_mtp,
  qwen3.5_fp8_mi325x_mtp). Patched their source paths from
  ../benchmark_lib.sh to ../../benchmark_lib.sh.
- runners/launch_mi355x-amds.sh: multinode-non-disagg BENCHMARK_SUBDIR
  bumped from `single_node` to `single_node/fixed_seq_len`.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

1 participant