vllm-omni-distributed

What problem does it solve?

Distributes large inference workloads across multiple GPUs or machines, enabling scalable deployments and efficient resource utilization.

Core Features & Use Cases

• Tensor Parallelism (TP): Split model weights across GPUs to reduce latency and improve throughput.
• Pipeline Parallelism (PP): Divide the model across sequential GPU groups to boost overall throughput.
• Disaggregation / OmniConnector: Run Encode, Prefill, Decode, and Generate stages on separate GPU pools for independent scaling.
• Multi-node with Ray: Orchestrate distributed serving across a Ray cluster for larger deployments.
• Sequence Parallelism: Accelerate diffusion-based generation by splitting steps across GPUs.

Quick Start

Launch a multi-node Ray cluster and start the vLLM-Omni server with your model and desired tensor- and pipeline-parallel settings.