AI 导读
针对打包多模态掩码的精确序列并行。 它逐层选择 ring attention、稀疏 all-to-all 或 Ulysses，数值与单 GPU 一致，在最多 128 块 H100 上，48 个案例中有 47 个优于 FlashMask + Ulysses 混合方案。 https://lumalabs.ai/news/packed-multimodal-attention
正文
Exact sequence parallelism for packed multimodal masks.
It picks ring attention, a sparse all-to-all or Ulysses per layer, matches single-GPU numerics, and beats a FlashMask + Ulysses hybrid in 47 of 48 cases on up to 128 H100s.
https://lumalabs.ai/news/packed-multimodal-attention
来源：Luma · x.com