skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

awq-quantization

Quantize LLMs to 4-bit precision using activation-aware weight scaling for GPU inference.

1|Updated Mar 5, 2026
Search Tags:#llm#quantization#model compression#4-bit#vllm#awq#gpu inference
One-click install
npx skills add https://github.com/Clay-HHK/claude-skills --skill awq-quantization-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: awq-quantization
Source: https://github.com/Clay-HHK/claude-skills/tree/main/awq
Command: npx skills add https://github.com/Clay-HHK/claude-skills --skill awq-quantization-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires autoawq, transformers, torch, and includes references (resource) components.

What problem does it solve? Deploying large language models (7B-70B) requires more GPU memory than most hardware provides, and naive quantization degrades accuracy. This Skill guides activation-aware 4-bit weight quantization that reduces memory footprint by up to 4x while keeping accuracy loss under 5%. ## Core Features & Use Cases - Model Quantization: Quantize models like Mistral, Llama, and Qwen to 4-bit using AutoAWQ with configurable group size, zero-point, and kernel versions (GEMM, GEMV, Marlin, ExLlama). - Deployment Integration: Load pre-quantized AWQ models through HuggingFace Transformers or serve them with vLLM for production inference. - Kernel & Hardware Selection: Choose between GEMM for batch inference, GEMV for single-token generation, Marlin for Ampere+ GPUs, and ExLlama for AMD hardware. - Use Case: You need to deploy a Mistral-7B chat model on a single RTX 4090. Quantize it with AWQ to shrink memory from 14 GB to 5.5 GB and serve it through vLLM with roughly 3x faster inference than FP16. ## Quick Start Quantize my Mistral-7B model to 4-bit using AWQ and show me how to load it with vLLM for inference.

Frequently Asked Questions about awq-quantization

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I quantize a Llama or Mistral model to 4-bit with AWQ?▼

Load the model with AutoAWQForCausalLM.from_pretrained, then call model.quantize with a config specifying w_bit=4, q_group_size=128, zero_point=True, and version GEMM. Save the result with save_quantized. Quantization takes about 10-15 minutes for a 7B model.

AWQ vs GPTQ vs bitsandbytes: which quantization should I use?▼

AWQ offers roughly 2.5-3x speedup with under 5% accuracy loss and low overfitting risk, making it best for production inference. GPTQ has broader ecosystem support, while bitsandbytes needs no calibration and suits QLoRA fine-tuning.

Does AWQ work with vLLM for model serving?▼

Yes, vLLM natively supports AWQ models. Pass quantization="awq" and dtype="half" when creating the LLM instance, and vLLM auto-detects the quantized weights for serving.

What GPUs are required for AWQ Marlin kernels?▼

Marlin kernels require NVIDIA GPUs with compute capability 8.0 or higher, such as A100, H100, or RTX 40xx series. AWQ generally requires compute capability 7.5+, so older GPUs like GTX 10xx and V100 are unsupported.

Why does AWQ quantization run out of CUDA memory?▼

Quantization loads the full model plus calibration data into GPU memory. Reduce max_calib_samples to 64, enable low_cpu_mem_usage, or distribute the model across multiple GPUs with device_map="auto".

Can AWQ quantize multimodal models like LLaVA?▼

Yes, AWQ supports multimodal architectures including LLaVA, LLaVA-Next, and Qwen2-VL. The vision encoder modules are typically excluded from quantization via modules_to_not_convert to preserve visual accuracy.

Related Skills

Explore More Software Engineering Skills→
NVIDIANVIDIA
2.8k

ptq

Quantize pretrained models to nvfp4, fp8, or int4_awq checkpoints using ModelOpt PTQ.

Official
Advanced
GitnappGitnapp

serving-llms-vllm

Deploy large language model inference services with vLLM and OpenAI-compatible endpoints.

Community
Advanced
OpenLAIROpenLAIR
1.0k

awq-quantization

Quantize large language models to 4-bit with activation-aware AWQ.

Official
Advanced
arsityarsity

gguf-quantization

Quantize machine learning models to GGUF for CPU and Apple Silicon deployment.

Community
Advanced