[NV] [PR3] feat(dsv4): enable synthetic-acceptance for dsv4-fp4-gb200 mtp2 sweep#2091
[NV] [PR3] feat(dsv4): enable synthetic-acceptance for dsv4-fp4-gb200 mtp2 sweep#2091qiching wants to merge 4 commits into
Conversation
Two changes needed to run dsv4 fp4 multinode sweeps on the watchtower (Oracle) gb200 cluster: - Point the dynamo-vllm dsv4 MODEL_PATH at the base DeepSeek-V4-Pro checkpoint instead of the -NVFP4 re-quant. The pinned v0.20.1 container's deepseek_v4 loader doesn't define the NVFP4 export's extra quant params (e.g. ffn.experts.w13_input_scale) and KeyErrors at load; the base checkpoint loads and matches the recipe's served identity. - Add dsv4/dynamo-vllm to uses_watchtower_shared_fs() so the srt-slurm workspace/outputs land on a compute-visible shared FS (the runner home is not cross-mounted to compute nodes, otherwise the batch step dies).
Framework-agnostic plumbing for optionally overriding a recipe's speculative-decoding acceptance length with a synthetic value, gated on SYNTHETIC_ACCEPTANCE=true (no-op otherwise, so this commit changes no existing behavior). - runners/synthetic_injectors/: backend registry (register/get_injector). No backends registered yet -> the generic layer alone injects nothing. - runners/inject_synthetic_acceptance.py: generic driver. Handles flag gating, acceptance-length resolution (explicit env or reference-yaml lookup) and file I/O, then delegates the recipe rewrite to the framework backend resolved from FRAMEWORK. Errors clearly if the flag is on but no backend is registered. - runners/launch_gb200-nv.sh: invoke the driver after the name override and before srtctl apply. With no backend registered this is a no-op.
Register a dynamo-vllm backend with the generic driver. When SYNTHETIC_ACCEPTANCE=true and FRAMEWORK=dynamo-vllm, every speculative-config JSON entry in the recipe is rewritten to add rejection_sample_method=synthetic and synthetic_acceptance_length=<al>, so the vLLM engine emits a controlled mean acceptance length instead of running the real draft model. Rewrite uses the json module on the captured JSON (no string-munging), preserving the recipe's compact style. - runners/synthetic_injectors/vllm.py: rewrite() + spec_tokens_from_recipe(), self-registers under "dynamo-vllm". - runners/synthetic_injectors/__init__.py: import the backend so it registers on package import. Still inert until a config opts in via SYNTHETIC_ACCEPTANCE (added for dsv4 mtp2 in a follow-up).
Opt the dsv4 fp4 gb200 dynamo-vllm mtp2 config into synthetic acceptance (length 2.27) on all four search-space cells, and add a matching -nosynthetic baseline config (identical topology/recipes, synthetic envs removed) so the sweep produces a before/after Pareto pair (throughput vs interactivity) in one run. This is the config that actually exercises the plumbing from the preceding PRs: base-checkpoint/shared-FS multinode fix, the generic injection driver, and the vLLM backend.
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
/sweep test-config --config-files configs/nvidia-master.yaml --config-keys dsv4-fp4-gb200-dynamo-vllm-mtp2 --no-evals |
1 participant
No description provided.