skills.rest
Orchestra-Research
Orchestra-ResearchOFFICIAL
Explore all skills by this author →
GitHub

quantizing-models-bitsandbytes

Quantizes LLMs to 8-bit or 4-bit formats for reduced GPU memory usage.

13.0k|930|Updated Nov 3, 2025
Search Tags:#llm#memory optimization#quantization#huggingface#bitsandbytes#4-bit#qlora
One-click install
npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill quantizing-models-bitsandbytes-orchestra-research
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: quantizing-models-bitsandbytes
Source: https://github.com/Orchestra-Research/AI-research-SKILLs/tree/main/10-optimization/bitsandbytes
Command: npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill quantizing-models-bitsandbytes-orchestra-research

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires bitsandbytes, transformers, accelerate, torch, and includes references (resource) components.

What problem does it solve? Large language models often exceed available GPU memory, blocking inference and fine-tuning on consumer or limited hardware. This Skill reduces model memory by 50-75% through 8-bit and 4-bit quantization with minimal accuracy loss. ## Core Features & Use Cases - 8-bit and 4-bit Quantization: Load models with INT8, NF4, or FP4 formats via HuggingFace BitsAndBytesConfig, cutting a 7B model from 14GB to as low as 3.5GB. - QLoRA Fine-tuning: Train LoRA adapters on 4-bit base models to fine-tune 70B models on a single GPU, saving only ~20MB adapter weights. - 8-bit Optimizers: Replace standard AdamW with paged 8-bit optimizers to reduce optimizer state memory by 75% during training. - Use Case: You have an RTX 4090 with 24GB VRAM and need to fine-tune Llama 2 13B. Load it in 4-bit NF4 with double quantization, attach LoRA adapters, and train with paged_adamw_8bit using roughly 18GB of memory. ## Quick Start Quantize the model meta-llama/Llama-2-7b-hf to 4-bit NF4 format and run a test generation to verify the memory reduction.

Frequently Asked Questions about quantizing-models-bitsandbytes

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I quantize a HuggingFace model to 4-bit?▼

Create a BitsAndBytesConfig with load_in_4bit=True, bnb_4bit_quant_type="nf4", and bnb_4bit_use_double_quant=True, then pass it as quantization_config to AutoModelForCausalLM.from_pretrained. This reduces memory by 75% compared to FP16.

What is the difference between NF4 and FP4 quantization?▼

NF4 uses quantization bins optimized for normally distributed weights, while FP4 uses symmetric bins for uniform distributions. NF4 gives better accuracy for transformer models and is the recommended default for LLMs.

How to fine-tune a 70B model on a single GPU with QLoRA?▼

Load the base model in 4-bit with NF4 and double quantization, prepare it with prepare_model_for_kbit_training, attach LoRA adapters via peft, and train with the paged_adamw_8bit optimizer. A 70B model fits in roughly 35GB plus small adapter weights.

Does bitsandbytes work with FSDP multi-GPU training?▼

Yes, but you must set bnb_4bit_quant_storage=torch.bfloat16 in the BitsAndBytesConfig so 4-bit layers are wrapped identically to regular layers for FSDP sharding. The torch_dtype passed to from_pretrained must match this storage dtype.

Why does quantization cause a CUDA error during model loading?▼

CUDA errors usually come from a bitsandbytes build that does not match your installed CUDA version. Check your version with nvcc --version and reinstall bitsandbytes with pip install bitsandbytes --no-cache-dir.

When should I use GPTQ or AWQ instead of bitsandbytes?▼

Use GPTQ or AWQ for production serving where inference speed matters more, since they run faster than bitsandbytes. Use GGUF for CPU inference with llama.cpp, and FP8 on H100 GPUs where hardware FP8 acceleration is available.

Related Skills

Explore More Software Engineering Skills→
projectedanxprojectedanx
1

fine-tuning-with-trl

Implement SFT, DPO, PPO, and GRPO post-training alignment with HuggingFace TRL.

Community
Advanced
cuba6112cuba6112

unsloth-gguf

Export fine-tuned machine learning models to GGUF format with quantization and LoRA merging.

Community
Intermediate
dongzhuoyaodongzhuoyao
10

gpu-training-acceleration

Optimize PyTorch training speed and memory usage on CUDA GPUs.

Community
Advanced
ehadziabdicehadziabdic

llm-deployment

Deploy and serve large language models with vLLM, TGI, Ollama, and Kubernetes.

Community
Advanced