skills.rest
Orchestra-Research
Orchestra-ResearchOFFICIAL
Explore all skills by this author →
GitHub

gguf-quantization

Convert and quantize HuggingFace models to GGUF format for llama.cpp inference.

13.0k|930|Updated Nov 3, 2025
Search Tags:#apple silicon#quantization#model compression#gguf#llama.cpp#cpu inference#llama-cpp-python
One-click install
npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill gguf-quantization-orchestra-research
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gguf-quantization
Source: https://github.com/Orchestra-Research/AI-research-SKILLs/tree/main/10-optimization/gguf
Command: npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill gguf-quantization-orchestra-research

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python>=0.2.0, and includes references (resource) components.

What problem does it solve? Running large language models on consumer hardware, CPUs, or Apple Silicon requires compressing models into an efficient format, and doing this manually involves complex conversion and quantization steps that are easy to get wrong. ## Core Features & Use Cases - Model Conversion: Convert HuggingFace models to GGUF format using llama.cpp's convert_hf_to_gguf.py script with FP16 or other output types. - Flexible Quantization: Apply 2-8 bit K-quant methods (Q2_K through Q8_0) with optional importance matrix (imatrix) calibration for better low-bit quality. - Multi-Platform Inference: Run quantized models via llama-cli, llama-server (OpenAI-compatible API), or llama-cpp-python on CPU, NVIDIA CUDA, and Apple Silicon Metal. - Use Case: Imagine you want to run Llama-3.1-8B on your MacBook. Use this Skill to convert the model to GGUF, quantize it to Q4_K_M with an importance matrix, and serve it locally through an OpenAI-compatible endpoint. ## Quick Start Convert my HuggingFace model to GGUF format and quantize it to Q4_K_M so I can run it locally with llama.cpp.

Frequently Asked Questions about gguf-quantization

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I convert a HuggingFace model to GGUF format?▼

Use llama.cpp's convert_hf_to_gguf.py script with the model path and output file arguments. Run python convert_hf_to_gguf.py ./model --outfile model-f16.gguf --outtype f16, then quantize the result with llama-quantize.

What GGUF quantization level should I use for a 7B model?▼

Q4_K_M is the recommended default, offering about 4.1 GB size with high quality for a 7B model. Use Q5_K_M or Q6_K for higher quality, or Q3_K_M when memory is severely constrained.

GGUF vs AWQ vs GPTQ: which quantization should I choose?▼

GGUF targets CPU, Apple Silicon, and flexible hardware deployment through llama.cpp. AWQ and GPTQ provide maximum accuracy with calibration on NVIDIA GPUs, while TensorRT-LLM suits production NVIDIA deployments requiring maximum speed.

Does GGUF work on Apple Silicon M1/M2/M3 chips?▼

Yes, GGUF supports Apple Silicon through Metal acceleration. Build llama.cpp with make GGML_METAL=1 and set n_gpu_layers to 99 in llama-cpp-python to offload all layers to the GPU.

Why does my quantized model produce poor quality output?▼

Low-bit quantization below Q4 often degrades quality without calibration. Generate an importance matrix with llama-imatrix using diverse calibration text, then pass it to llama-quantize with the --imatrix flag.

How do I fix CUDA out of memory errors when running GGUF models?▼

Reduce the number of GPU offloaded layers by lowering n_gpu_layers, switch to a smaller quantization like Q3_K_M, reduce context length, or quantize the KV cache using type_k and type_v parameters.

Related Skills

Explore More Software Engineering Skills→
jinzhezenggroupjinzhezenggroup
124

deepmd-finetune-dpa3

Fine-tunes a pre-trained DPA3 model on a downstream dataset using DeePMD-kit.

Official
Advanced
azaanalirazaazaanaliraza

llama-cpp

Run and quantize GGUF LLMs with llama.cpp across CPU, Apple Silicon, and GPU hardware.

Community
Advanced
Cloud-Byte-ConsultingCloud-Byte-Consulting

inference-benchmark-runner

Benchmarks LLM inference latency, throughput, GPU utilization, and cost on Kubernetes.

Official
Advanced
hung-phanhung-phan
1

quantization

Quantize pre-trained Large Language Models using PTQ, QAT, GPTQ, AWQ, bitsandbytes, FP8, and GGUF.

Community
Advanced