dsv4 sglang agentic: enable out-of-window SWA slot release and bump image to nightly-20260707#2112
Conversation
…ge to nightly-20260707 Three changes: 1/2. dsv4_fp4_b200_sglang.sh + dsv4_fp4_b300_sglang.sh: unconditionally export SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 and SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 (applies to the hicache jobs; the previously hicache-only unified-tree export is folded in). 3. dsv4-fp4-b200/b300-sglang-agentic-hicache image: v0.5.13-cu130 -> nightly-dev-cu13-20260707-b4155233. Rationale: v0.5.13 predates both sgl-project/sglang#27402 (the slot-release feature, 2026-06-11) and sgl-project/sglang#29860 (fix for SWA eviction tombstoning the last leaf under that flag, 2026-07-02). The 20260707 nightly contains both. Why: with radix cache on, chunked prefill keeps each in-flight request whole- context SWA KV cache-protected, oversubscribing the SWA partition ~5x at conc>=32; allocation then silently evicts idle sessions cached windows and prefix-cache hits fail all-or-nothing. Measured on this repo agentic bench (8xB200, conc32/600s, GPU-only, sglang main 2026-07-04): actual hit 44.5% -> 88.5%, 40.7K -> ~150K tok/s, TTFT 16.8s -> 1.9s, evict storm 3182 -> 0. Correctness: GSM8K 0.970 (200q), long-prefix GSM8K 0.950 (60q, 12K filler), needle-in-haystack exact recall, output-length distributions unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cquil11
requested review from
Ankur-singh,
jgangani and
kedarpotdar-nv
as code owners
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
|
||
| dsv4-fp4-b200-sglang-agentic-hicache: | ||
| image: lmsysorg/sglang:v0.5.13-cu130 | ||
| image: lmsysorg/sglang:nightly-dev-cu13-20260707-b4155233 |
There was a problem hiding this comment.
🔴 This PR bumps the SGLang image tag for dsv4-fp4-b200-sglang-agentic-hicache (line 12683) and dsv4-fp4-b300-sglang-agentic-hicache (line 12705) but does not append a matching entry to perf-changelog.yaml. Per AGENTS.md's "Updating Docker images" section, a changelog entry is required — it's what triggers .github/workflows/run-sweep.yml (which filters on paths: - perf-changelog.yaml). Without it, neither the PR fail-fast sweep nor the post-merge push-to-main sweep will actually run for these two configs, so the ~40K→106K tok/s / TTFT 17s→6s improvements claimed in the PR description will not be reflected in tracked main-branch results.
Extended reasoning...
What the bug is
The PR modifies exactly 3 files: two launcher scripts (benchmarks/single_node/agentic/dsv4_fp4_b{200,300}_sglang.sh) and configs/nvidia-master.yaml. In configs/nvidia-master.yaml it bumps the image for two configs:
dsv4-fp4-b200-sglang-agentic-hicacheat line 12683:lmsysorg/sglang:v0.5.13-cu130→lmsysorg/sglang:nightly-dev-cu13-20260707-b4155233dsv4-fp4-b300-sglang-agentic-hicacheat line 12705: same bump
It also adds two new engine env vars (SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 and SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1) to both launcher scripts. However, perf-changelog.yaml is not touched at all.
Why this matters — the trigger contract
AGENTS.md (Updating Docker images) is explicit: on an image bump you must "append a perf-changelog.yaml entry (required - triggers benchmarks)". This is not just documentation hygiene — it is the actual trigger:
# .github/workflows/run-sweep.yml
on:
push:
branches: [main]
paths:
- perf-changelog.yaml
pull_request:
paths:
- perf-changelog.yamlBecause the workflow's paths: filter only fires when perf-changelog.yaml is modified, this PR's diff will not cause run-sweep.yml to invoke — despite the full-sweep-fail-fast label being set. The label controls behavior inside the workflow once it runs; it does not cause the workflow to run in the first place.
Concrete step-by-step proof
- Contributor pushes commits touching
configs/nvidia-master.yamland the two.shfiles. - GitHub Actions receives the
pull_requestevent. - Actions consults
run-sweep.yml'son.pull_request.pathsfilter:perf-changelog.yaml— not in the changed file list. - Actions does not schedule the sweep workflow. No matrix job runs; no fail-fast gate exists.
- PR merges (nothing gated it).
pushtomainfires;on.push.pathsfilter again requiresperf-changelog.yaml— again not present. - No post-merge sweep runs against these two configs on the new image. The tracked main-branch performance dashboards keep the old image's numbers.
Precedent — every recent bump followed this convention
Recent image-bump PRs (#2056, #2058, #2060, #2067, #2073, #2074, #2077) all touched both configs/*-master.yaml and appended a perf-changelog.yaml block. That's the workflow this repo relies on.
Fix
Append an entry to perf-changelog.yaml in the same style as the tail of that file — model+precision+SKU, PR number, one-line reason (e.g. "Bump sglang nightly to pick up unified radix-tree out-of-window SWA slot release"), and the two affected config keys (dsv4-fp4-b200-sglang-agentic-hicache and dsv4-fp4-b300-sglang-agentic-hicache). That single addition arms both the PR fail-fast sweep and the post-merge push-to-main sweep.
cquil11
force-pushed
the
reopen/pr-2111-sglang-swa
branch
from
48cc2e4 to
be39b9f
Compare
… declared fp4 The launcher preferred a node-local /raid NVFP4 checkpoint for dsv4+fp4+sglang while the config declares fp4 (deepseek-ai/DeepSeek-V4-Pro). This silently made sglang benchmark NVFP4 while vLLM benchmarked the declared fp4 (an undeclared, unfair mismatch), and it crashed the flashinfer_mxfp4 low-conc path since the ModelOpt NVFP4 MoE has no fused func registered for it. Remove the swap so sglang runs the declared fp4 checkpoint, matching vLLM and the yaml.
|
@claude add a perf-changelog entry to this pr. dont make it too verbose, it should be brief yet hold all information necessary to understand what changed |
|
Claude finished @cquil11's task in 56s —— View job Perf-changelog entry added
Added a changelog entry (commit
Since the PR already exists, I used the real PR link rather than the |
…nv vars Co-authored-by: Cameron Quilici <60715037+cquil11@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28955639528 |
2 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28955639528 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28955639528 |
|
/reuse-sweep-run 28955639528 |
3 participants
Change
1. Update the SGLang image to a recent nightly
lmsysorg/sglang:v0.5.13-cu130→lmsysorg/sglang:nightly-dev-cu13-20260707-b4155233for the
dsv4-fp4-b200-sglang-agentic-hicacheanddsv4-fp4-b300-sglang-agentic-hicachejobs. The newer build carries recent kernel and runtime updates
2. Enable two engine env vars in the b200/b300 agentic launch scripts
SGLANG_ENABLE_UNIFIED_RADIX_TREE=1— selects the unified radix tree, the cacheimplementation that supports per-component (full-attention / SWA) management for
hybrid-attention models. It was previously exported only inside the hicache branch;
it is now set unconditionally (the duplicate in-branch export is removed).
SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1— proactively freesout-of-window SWA KV slots during chunked prefill. Without it, in-flight
requests hold SWA KV for their entire context, which keeps the SWA pool under
constant pressure and forces frequent evictions. Most importantly, under LRU
the last-window KV of cached sessions gets flushed as well; since a prefix
match is only valid when that trailing window is present, hits become
bimodal — a session either matches almost fully or not at all — and the
effective prefix-cache hit rate collapses on multi-turn agentic workloads.
Enabling the release removes this pressure at the source.
Perf Comparison
Measured on an 8×B200 single-node reproduction of this agentic benchmark at concurrency 64 DEP =8, No Hicache (600 s trace replay):
Note: reopening this directly rather than from a fork. Original author: @Oasis-Git