What problem does it solve? Training large language models with dense architectures requires compute that scales linearly with parameter count, making large-capacity models prohibitively expensive. This Skill guides training of Mixture of Experts (MoE) models that activate only a fraction of parameters per token, achieving roughly 5× training cost reduction compared to dense equivalents. ## Core Features & Use Cases - MoE Architecture Implementation: Build sparse MoE layers with top-k routing, gating networks, load balancing losses, and router z-loss for training stability. - DeepSpeed & Megatron-DeepSpeed Training: Configure expert parallelism, capacity factors, PR-MoE pyramid structures, and Mixture-of-Students distillation for large-scale pretraining. - Inference Optimization: Deploy MoE models with vLLM using FP8 quantization, fused MoE kernels, speculative decoding, and expert pruning. - Use Case: Reproduce a Mixtral 8x7B-style model with 8 experts and top-2 routing across 8 GPUs, distributing experts with expert parallelism and tuning the auxiliary loss coefficient to prevent load imbalance. ## Quick Start Ask the AI to help you configure and launch a DeepSpeed MoE training run with a specified number of experts, GPUs, and dataset path.
