skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

quantizing-models-bitsandbytes

Quantizes LLMs to 8-bit or 4-bit formats for reduced GPU memory usage.

1|Updated Mar 5, 2026
Search Tags:#llm#fine-tuning#memory optimization#quantization#bitsandbytes#4-bit#qlora
One-click install
npx skills add https://github.com/Clay-HHK/claude-skills --skill quantizing-models-bitsandbytes-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: quantizing-models-bitsandbytes
Source: https://github.com/Clay-HHK/claude-skills/tree/main/bitsandbytes
Command: npx skills add https://github.com/Clay-HHK/claude-skills --skill quantizing-models-bitsandbytes-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires bitsandbytes, transformers, accelerate, torch, and includes references (resource) components.

What problem does it solve? Large language models often exceed available GPU memory, blocking inference and fine-tuning on consumer hardware. This Skill reduces model memory by 50-75% through 8-bit and 4-bit quantization with minimal accuracy loss. ## Core Features & Use Cases - 8-bit and 4-bit Quantization: Load models with INT8, NF4, or FP4 formats via HuggingFace BitsAndBytesConfig, cutting a 14GB Llama 2 7B down to 3.5GB. - QLoRA Fine-tuning: Train LoRA adapters on 4-bit base models to fine-tune 70B models on a single GPU, saving only ~20MB adapter weights. - 8-bit Optimizers: Replace standard AdamW with paged 8-bit optimizers to reduce optimizer state memory by 75%. - Use Case: You have an RTX 4090 with 24GB VRAM and need to fine-tune Llama 2 13B. Load it in 4-bit with NF4 and double quantization, attach LoRA adapters, and train with the paged_adamw_8bit optimizer using roughly 18GB of memory. ## Quick Start Quantize the model meta-llama/Llama-2-7b-hf to 4-bit with NF4 and double quantization, then run a test generation to verify memory usage.

Frequently Asked Questions about quantizing-models-bitsandbytes

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I quantize a HuggingFace model to 4-bit?▼

Create a BitsAndBytesConfig with load_in_4bit=True, bnb_4bit_quant_type="nf4", and bnb_4bit_use_double_quant=True, then pass it as quantization_config to AutoModelForCausalLM.from_pretrained. A 7B model drops from 14GB to about 3.5GB.

What is QLoRA and how do I fine-tune with it?▼

QLoRA fine-tunes a 4-bit quantized base model by training small LoRA adapters instead of full weights. Load the model in 4-bit, call prepare_model_for_kbit_training, attach adapters with peft's get_peft_model, and train with a standard Trainer using paged_adamw_8bit.

bitsandbytes vs GPTQ vs GGUF: which should I use?▼

Use bitsandbytes for QLoRA training and quick HuggingFace integration with 50-75% memory savings. Choose GPTQ or AWQ for faster production inference serving, and GGUF for CPU inference with llama.cpp.

Does bitsandbytes work on AMD or non-NVIDIA GPUs?▼

bitsandbytes primarily targets NVIDIA GPUs with compute capability 7.0+ (Turing, Ampere, Hopper) and CUDA 11.1+. AMD ROCm and Intel GPU support exist but are experimental.

Why do I get CUDA errors when loading a quantized model?▼

CUDA errors usually come from a version mismatch between bitsandbytes and your installed CUDA toolkit. Check your version with nvcc --version and reinstall bitsandbytes with pip install bitsandbytes --no-cache-dir.

What should I do if I still get OOM with 4-bit quantization?▼

Enable CPU offloading by setting max_memory={0: "20GB", "cpu": "100GB"} in from_pretrained, or add disk offloading with offload_folder. For training, also enable gradient checkpointing and reduce batch size to 1.

Related Skills

Explore More Software Engineering Skills→
qcmuuqcmuu

peft-fine-tuning

Train LoRA or QLoRA adapters for parameter-efficient fine-tuning of transformer models.

Community
Intermediate
jaredcroxtonjaredcroxton

serving-llms-vllm

Deploy OpenAI-compatible LLM inference servers with vLLM and quantization.

Community
Intermediate
HermeticOrmusHermeticOrmus

Distributed Training Patterns

Implement distributed training patterns with PyTorch DDP, FSDP, and DeepSpeed ZeRO-3.

Community
Advanced
hung-phanhung-phan
1

quantization

Quantize pre-trained Large Language Models using PTQ, QAT, GPTQ, AWQ, bitsandbytes, FP8, and GGUF.

Community
Advanced