skills.rest
Orchestra-Research
Orchestra-ResearchOFFICIAL
Explore all skills by this author →
GitHub

gptq

Quantize LLMs to 4-bit precision using group-wise post-training quantization.

13.0k|930|Updated Nov 3, 2025
Search Tags:#llm#quantization#inference#model compression#4-bit#gptq#qlora
One-click install
npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill gptq-orchestra-research
Or copy as Structured Prompt for Agentâ–¼
Please help me install this Agent Skill.
Skill: gptq
Source: https://github.com/Orchestra-Research/AI-research-SKILLs/tree/main/10-optimization/gptq
Command: npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill gptq-orchestra-research

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires auto-gptq, transformers, optimum, peft, and includes references (resource) components.

What problem does it solve? Large language models like Llama 2-70B require 140GB of GPU memory in FP16, making them impossible to run on consumer hardware or single GPUs. GPTQ compresses these models to 4-bit precision, reducing memory by 4× with under 2% perplexity degradation while accelerating inference 3-4×. ## Core Features & Use Cases - Post-Training Quantization: Quantize any supported model (Llama, Mistral, Qwen, DeepSeek) to 4-bit using calibration data and group-wise quantization with configurable group sizes. - Multiple Kernel Backends: Run quantized models with ExLlamaV2, Marlin (Ampere+ GPUs), or Triton backends for optimized inference speed. - QLoRA Fine-Tuning: Combine GPTQ models with PEFT LoRA adapters to fine-tune 70B models on a single A100 80GB GPU. - Use Case: Deploy Llama 2-70B on a single A100 by loading a pre-quantized GPTQ model from HuggingFace, or quantize your own fine-tuned model with 128 C4 calibration samples for production serving via vLLM. ## Quick Start Quantize my Llama 2-7B model to 4-bit using GPTQ with group size 128 and C4 calibration data, then save it for deployment.

Frequently Asked Questions about gptq

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I quantize a Llama model to 4-bit with GPTQ?â–¼

Load the model with AutoGPTQForCausalLM using a BaseQuantizeConfig with bits=4 and group_size=128, then call model.quantize() with 128 calibration samples of 512 tokens from the C4 dataset. Save the result with save_quantized().

GPTQ vs AWQ vs bitsandbytes for LLM quantization?â–¼

GPTQ offers 4× memory reduction with under 2% accuracy loss and broad model support. AWQ gives slightly better accuracy and Marlin kernel support on newer GPUs. bitsandbytes provides simpler transformers integration and 8-bit quantization with better quality but less compression.

Can I fine-tune a GPTQ quantized model with LoRA?â–¼

Yes, load the GPTQ model with transformers, apply prepare_model_for_kbit_training, then attach LoRA adapters via PEFT's get_peft_model. This QLoRA approach enables fine-tuning a 70B model on a single A100 80GB GPU.

What group size should I use for GPTQ quantization?â–¼

Group size 128 is the recommended default, balancing accuracy, model size, and inference speed. Use 32 for maximum accuracy at slightly larger size, or 256 when speed is critical. Per-column (-1) is suited only for research.

Why is my GPTQ model producing poor quality outputs?â–¼

Poor quality usually stems from inadequate calibration data. Use 128-256 domain-matched samples, lower damp_percent to 0.005, enable desc_act=True, or reduce group_size to 32. Perplexity increases above 5% indicate recalibration is needed.

Does GPTQ work with vLLM for inference serving?â–¼

Yes, vLLM supports GPTQ models by passing quantization="gptq" when initializing the LLM class. Text Generation Inference also supports GPTQ via the --quantize gptq flag in its Docker deployment.

Related Skills

Explore More Software Engineering Skills→
fanguyunfanguyun

fine-tuning-expert

Automates LLM fine-tuning with LoRA/QLoRA, dataset prep, hyperparameters, and deployment optimization.

Community
Advanced
jaredcroxtonjaredcroxton

serving-llms-vllm

Deploy OpenAI-compatible LLM inference servers with vLLM and quantization.

Community
Intermediate
DoanNgocCuongDoanNgocCuong

evaluating-llms-harness

Evaluate LLMs against over 60 academic benchmarks like MMLU and GSM8K.

Community
Advanced
Ced3-hanCed3-han

gguf-quantization

Convert HuggingFace models to GGUF format and quantize them for llama.cpp inference.

Community
Advanced