Skip to content

lingbot-video-moe-30b-a3b_ti2v.sh多机训练问题,每个机器各跑个的,没有并行 #1744

Open

Description

@chenhangcal

accelerate_config_moe.yaml配置修改如下,其他没有修改,直接运行样例脚本lingbot-video-moe-30b-a3b_ti2v.sh,没有并行起来

compute_environment: LOCAL_MACHINE
debug: false
deepspeed_config:
gradient_accumulation_steps: 1
offload_optimizer_device: none
offload_param_device: none
zero3_init_flag: true
zero_stage: 3
zero3_save_16bit_model: true
distributed_type: DEEPSPEED
downcast_bf16: 'no'
enable_cpu_affinity: true
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 4
num_processes: 32
main_process_ip: 192.168.16.46
main_process_port: 29500
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false

Activity

  1. Artiprocher commented on Oct 14, 2026

    @Artiprocher
    Collaborator

    @chenhangcal 多机训练的配置比较繁琐,请确认:

    • 使用 accelerate config 进行配置
    • 确保多台机器之间的通信是正常的,开启 RDMA 通讯
    • 每一台机器启动时的环境变量配置不同,machine_rank 分别为 0,1,2...,否则每台机器都会把自己当作主机器
    • 如果 Deepspeed ZeRO3 不兼容,考虑关闭 DeepSpeed
  2. Artiprocher commented on Oct 14, 2026

    @Artiprocher
    Collaborator
  3. chenhangcal commented on Oct 14, 2026

    @chenhangcal
    Author

    用accelerate config配置了,感觉是accelerate的多机配置没有传递到DeepSpeed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions