skills.rest
kaminocorp
kaminocorpOFFICIAL
Explore all skills by this author →
Website𝕏@kaminocorpoGitHub

gguf-quantization

Convert and quantize HuggingFace models to GGUF format for llama.cpp inference.

1|Updated Mar 12, 2026
Search Tags:#apple silicon#quantization#gguf#llama.cpp#llm deployment#cpu inference#model conversion
One-click install
npx skills add https://github.com/kaminocorp/hermes-alpha-hunter --skill gguf-quantization-kaminocorp
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gguf-quantization
Source: https://github.com/kaminocorp/hermes-alpha-hunter/tree/main/skills/mlops/inference/gguf
Command: npx skills add https://github.com/kaminocorp/hermes-alpha-hunter --skill gguf-quantization-kaminocorp

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python, and includes references (resource) components.

What problem does it solve? Deploying large language models on consumer hardware, CPUs, or Apple Silicon requires converting models to an efficient format and compressing them to fit limited memory, which involves complex tooling and quantization trade-offs. ## Core Features & Use Cases - Model Conversion: Convert HuggingFace models to GGUF format using llama.cpp's conversion scripts with FP16 or other output types. - Flexible Quantization: Apply 2-8 bit K-quant methods (Q2_K through Q8_0) with optional importance matrix (imatrix) calibration for better low-bit quality. - Multi-Platform Inference: Run quantized models via llama-cli, llama-cpp-python, or an OpenAI-compatible server on CPU, NVIDIA CUDA, or Apple Silicon Metal. - Use Case: Take a Llama-3.1-8B model from HuggingFace, convert it to GGUF, quantize it to Q4_K_M with an importance matrix, and serve it through an OpenAI-compatible API on a laptop with 8GB VRAM. ## Quick Start Convert my HuggingFace model to GGUF format and quantize it to Q4_K_M so I can run it locally with llama.cpp.

Frequently Asked Questions about gguf-quantization

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I convert a HuggingFace model to GGUF format?▼

Use the convert_hf_to_gguf.py script from llama.cpp: run python convert_hf_to_gguf.py ./model --outfile model-f16.gguf --outtype f16. Then quantize the FP16 output with llama-quantize to your target bit level such as Q4_K_M.

What GGUF quantization level should I use for a 7B model?▼

Q4_K_M is the recommended default, offering about 4.1 GB size with high quality for a 7B model. Use Q5_K_M or Q6_K for higher quality, or Q3_K_M when memory is severely constrained.

GGUF vs AWQ vs GPTQ: which quantization should I choose?▼

GGUF targets CPU, Apple Silicon, and flexible hardware deployment through llama.cpp with no Python runtime needed. AWQ and GPTQ are better for maximum accuracy on NVIDIA GPUs with calibration, while GGUF excels at cross-platform local inference.

Does GGUF work on Apple Silicon Macs?▼

Yes, GGUF runs on Apple Silicon with Metal acceleration. Build llama.cpp with make GGML_METAL=1 and set n_gpu_layers to 99 to offload all layers to the GPU via Metal.

Why does my quantized model produce poor quality output?▼

Low-bit quantization without calibration degrades quality. Generate an importance matrix with llama-imatrix using diverse calibration text, then pass it to llama-quantize with the --imatrix flag, especially for Q4 and below.

How do I fix CUDA out of memory errors with GGUF models?▼

Reduce the number of GPU offloaded layers by lowering n_gpu_layers, use a smaller quantization like Q3_K_M, reduce context length, or quantize the KV cache with type_k and type_v settings.

Related Skills

Explore More Software Engineering Skills→
thinking-machines-labthinking-machines-lab
4.0k

weights

Downloads, merges LoRA adapters, and publishes models to HuggingFace Hub.

Official
Intermediate
AVOI-CEOAVOI-CEO

llama-cpp

Run local LLM inference with llama.cpp across CPU, Apple Silicon, and GPUs.

Community
Advanced
qcmuuqcmuu

gptq

Quantize large language models to 4-bit with GPTQ calibration.

Community
Intermediate
handsomelong922handsomelong922

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced