vllm-omni-quantization

What problem does it solve?

Quantization reduces model memory footprint and increases inference throughput for vLLM-Omni, enabling efficient deployment on GPUs with limited VRAM.

Core Features & Use Cases

• Supports AWQ, GPTQ, and FP8 weight quantization to save memory and speed up autoregressive decoding.
• Guidance for serving pre-quantized models and selecting appropriate quantization modes based on hardware.
• Real-world usage includes fitting larger Omni models on fewer GPUs and lowering serving costs.

Quick Start

Quantize a base model with AWQ or GPTQ and start serving with the appropriate --quantization flag.