skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

hqq-quantization

Quantize LLM weights to 4/3/2-bit precision without calibration data using HQQ.

1|Updated Mar 5, 2026
Search Tags:#llm#quantization#model compression#huggingface#vllm#inference optimization#hqq
One-click install
npx skills add https://github.com/Clay-HHK/claude-skills --skill hqq-quantization-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: hqq-quantization
Source: https://github.com/Clay-HHK/claude-skills/tree/main/hqq
Command: npx skills add https://github.com/Clay-HHK/claude-skills --skill hqq-quantization-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires hqq, torch, and includes references (resource) components.

What problem does it solve? Quantizing large language models typically requires calibration datasets and hours of processing with methods like GPTQ or AWQ. This Skill enables fast, calibration-free weight quantization to 8/4/3/2/1-bit precision, reducing model memory footprint for deployment on limited GPU hardware. ## Core Features & Use Cases - Calibration-Free Quantization: Quantize any model in minutes using HqqConfig with HuggingFace Transformers, no sample data required. - Multiple Inference Backends: Switch between PyTorch, ATEN, TorchAO, Marlin, and BitBlas backends for optimized inference on different GPU hardware. - Framework Integration: Deploy quantized models with vLLM, fine-tune them with LoRA/PEFT, and save or push quantized models to the HuggingFace Hub. - Use Case: Quantize Llama-3.1-8B to 4-bit with group_size=64, serve it through vLLM with quantization="hqq", then apply LoRA adapters for domain-specific fine-tuning. ## Quick Start Quantize the model meta-llama/Llama-3.1-8B to 4-bit precision with HQQ and verify the generation quality.

Frequently Asked Questions about hqq-quantization

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I quantize a HuggingFace model to 4-bit with HQQ?▼

Pass a HqqConfig with nbits=4 and group_size=64 to AutoModelForCausalLM.from_pretrained as the quantization_config argument. The model quantizes on load without any calibration data and can be saved with save_pretrained.

HQQ vs GPTQ vs AWQ for LLM quantization?▼

HQQ requires no calibration data and quantizes in minutes, while GPTQ and AWQ use calibration datasets for potentially higher accuracy at the cost of longer processing. Choose HQQ for fast workflows, GPTQ or AWQ when calibration data is available and maximum accuracy matters.

Can I serve HQQ quantized models with vLLM?▼

Yes, vLLM supports HQQ quantization. Load a pre-quantized model by passing quantization="hqq" to the LLM constructor, or quantize on load by providing a quantization_config with nbits and group_size.

Can I fine-tune an HQQ quantized model with LoRA?▼

Yes, HQQ models are PEFT compatible. Load the quantized model, call prepare_model_for_kbit_training, then apply a LoraConfig targeting attention projections with get_peft_model and train normally with the HuggingFace Trainer.

Why is HQQ inference slow after quantization?▼

The default PyTorch backend is slowest. Switch to an optimized backend with HQQLinear.set_backend, using marlin for Ampere or newer GPUs, torchao_int4 for 4-bit inference, or apply torch.compile for additional speedup.

How do I fix poor quality with 2-bit HQQ quantization?▼

Reduce the group_size to 16 or 32 for finer-grained quantization at low bit-widths. You can also keep sensitive layers like attention projections at 4-bit while quantizing MLP layers to 2-bit with mixed precision configs.

Related Skills

Explore More Software Engineering Skills→
GregsGreyCodeGregsGreyCode
6

distributed-llm-pretraining-torchtitan

Configures and launches 4D-parallel LLM pretraining on multi-node GPUs.

Community
Advanced
InverterNetworkInverterNetwork

serving-llms-vllm

Deploy vLLM inference servers with OpenAI-compatible endpoints and quantization.

Official
Advanced
kaminocorpkaminocorp
1

evaluating-llms-harness

Benchmarks LLMs across 60+ academic tasks using standardized prompts and metrics.

Official
Advanced
helix4uhelix4u

gguf-quantization

Quantize machine learning models into GGUF format for llama.cpp inference.

Community
Intermediate