Prime Intellect Releases Training Kernel for MoE Models
AI · First reported 29 Oct, 17:41 · Updated 19:46
Prime Mega MoE adds an overlapping backward pass to DeepSeek’s fused MoE kernel and integrates it into Prime-RL. Prime Intellect reports faster training steps on two Qwen3 MoE models, with results measured on H200 GPUs.
- Prime Mega MoE adds a fused, overlapping backward pass to DeepSeek’s Mega MoE kernel, making it usable for training as well as inference.
- Prime Intellect reports full-step training gains of 32–34% over its Prime-RL baseline and 48–55% over TorchTitan on the tested Qwen3 models.
- The reports distinguish kernel-level speedups from end-to-end training results; the benchmarks used H200 GPUs, and the reported gains may vary across workloads and setups.
Summary written by Lux AI from 2 articles
Key facts
- Full training-step gain vs Prime-RL baseline
- 32–34%, as reported by Prime Intellect.
- Full training-step gain vs TorchTitan
- 48–55% on the reported Qwen3 models.
- Largest reported forward-pass speedup
- 3.11× on a Qwen3-30B-A3B MoE layer.
- Reported benchmark GPUs
- H200 GPUs, with tests using 32 or 128 GPUs.
- Supported formats and hardware
- BF16 on Hopper and Blackwell; blockwise FP8 on Hopper; MXFP8 on Blackwell.
Source spectrum
Far left
Left
Lean left
Centre
Lean right
Right
Far right
Unrated
Left 0 · Centre 1 · Right 0 of 1 rated sources · +1 unrated
All coverage
Coverage over time
LeftCentreRight
Articles per hour since first reported, by side.
Ownership
- Corporation 50% (1)
- Unknown 50% (1)
Share of the 2 sources by owner type.