torch-tensor-parallelism

What problem does it solve?

This Skill addresses the complexities of implementing and debugging tensor parallelism in PyTorch, specifically for column-parallel and row-parallel linear layers, enabling the training of larger models.

Core Features & Use Cases

• Tensor Parallelism Implementation: Provides guidance for ColumnParallelLinear and RowParallelLinear classes.
• Distributed Training Support: Essential for splitting weights and activations across multiple devices/processes using torch.distributed.
• Use Case: When building large language models that exceed single-GPU memory, this skill helps correctly shard model weights and manage communication for efficient distributed training.

Quick Start

Implement the ColumnParallelLinear class in PyTorch following the provided guidance.