skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

gptq

Quantize LLMs to 4-bit precision using group-wise post-training quantization.

1|Updated Mar 5, 2026
Search Tags:#llm#quantization#inference#model compression#4-bit#gptq#qlora
One-click install
npx skills add https://github.com/Clay-HHK/claude-skills --skill gptq-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gptq
Source: https://github.com/Clay-HHK/claude-skills/tree/main/gptq
Command: npx skills add https://github.com/Clay-HHK/claude-skills --skill gptq-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires auto-gptq, transformers, optimum, peft, and includes references (resource) components.

What problem does it solve? Large language models like Llama 2-70B require 140GB of GPU memory in FP16, making them impossible to run on consumer hardware. GPTQ compresses these models to 4-bit precision, achieving 4× memory reduction with under 2% perplexity degradation and 3-4× faster inference. ## Core Features & Use Cases - Post-Training Quantization: Quantize any supported model to 4-bit using calibration data and group-wise quantization with configurable group sizes (32, 128, 256). - Multiple Kernel Backends: Run inference with ExLlamaV2, Marlin (Ampere+ GPUs), or Triton for optimized speed on different hardware. - QLoRA Fine-Tuning: Combine GPTQ models with PEFT LoRA adapters to fine-tune 70B models on a single A100 80GB GPU. - Use Case: Deploy Llama 2-70B on a single A100 by loading a pre-quantized GPTQ model from HuggingFace, reducing memory from 140GB to 35GB while maintaining near-FP16 accuracy. ## Quick Start Load the pre-quantized model TheBloke/Llama-2-7B-Chat-GPTQ with AutoGPTQForCausalLM and generate a response to a test prompt.

Frequently Asked Questions about gptq

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I quantize a Llama model to 4-bit with GPTQ?▼

Load the model with AutoGPTQForCausalLM using a BaseQuantizeConfig set to 4 bits and group_size 128, then call model.quantize() with 128 calibration samples of 512 tokens from the C4 dataset. Save the result with save_quantized().

GPTQ vs AWQ vs bitsandbytes for LLM quantization?▼

GPTQ offers 4× memory reduction with under 2% accuracy loss and broad model support. AWQ gives slightly better accuracy and Marlin kernel support on newer GPUs, while bitsandbytes provides simpler transformers integration with 8-bit quantization.

Can I fine-tune a GPTQ quantized model with LoRA?▼

Yes, load the GPTQ model through transformers, apply prepare_model_for_kbit_training from PEFT, then attach a LoraConfig targeting modules like q_proj and v_proj. This enables QLoRA fine-tuning of 70B models on a single A100 80GB.

What group size should I use for GPTQ quantization?▼

Group size 128 is the recommended default, balancing accuracy, model size, and inference speed. Use 32 for higher accuracy at slightly larger size, or 256 when inference speed is critical.

Why is my GPTQ model producing poor quality output?▼

Poor quality usually stems from inadequate calibration data. Use 128-256 domain-matched samples, lower damp_percent to 0.005, enable desc_act=True, or try a smaller group size like 32 to reduce perplexity degradation.

Which GPUs support Marlin kernels for GPTQ inference?▼

Marlin requires NVIDIA Ampere or newer GPUs with compute capability 8.0 or higher, including A100, H100, and RTX 40-series cards. It delivers roughly 2× faster inference compared to standard CUDA kernels on supported hardware.

Related Skills

Explore More Software Engineering Skills→
box755box755

nanogpt

Trains GPT-style language models with PyTorch on Shakespeare or OpenWebText data.

Community
Intermediate
anxiety135790anxiety135790

serving-llms-vllm

Deploy and optimize vLLM serving with PagedAttention and continuous batching.

Community
Advanced
cabeznocabezno

evaluating-llms-harness

Benchmark LLMs on standardized tasks using the lm-evaluation-harness CLI.

Community
Intermediate
ovachieverovachiever
52

quantizing-models-bitsandbytes

Quantize LLMs to 8-bit or 4-bit precision using BitsAndBytes.

Community
Advanced