skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

llama-cpp

Runs quantized GGUF LLM inference on CPUs, Apple Silicon, and non-NVIDIA GPUs.

1|Updated Mar 5, 2026
Search Tags:#apple silicon#quantization#edge deployment#gguf#llama.cpp#llm serving#cpu inference
One-click install
npx skills add https://github.com/Clay-HHK/claude-skills --skill llama-cpp-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/Clay-HHK/claude-skills/tree/main/llama-cpp
Command: npx skills add https://github.com/Clay-HHK/claude-skills --skill llama-cpp-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python, and includes references (resource) components.

What problem does it solve? Running large language models typically requires expensive NVIDIA GPUs and CUDA infrastructure, leaving CPU-only machines, Apple Silicon Macs, AMD/Intel GPUs, and edge devices without a viable inference path. This Skill provides guidance for deploying LLMs on commodity hardware using llama.cpp with GGUF quantization. ## Core Features & Use Cases - Cross-Platform Inference: Run LLMs on CPU-only servers, Apple Silicon via Metal, AMD GPUs via ROCm, and NVIDIA GPUs via CUDA, with hybrid CPU+GPU layer offloading. - GGUF Quantization Guidance: Choose among Q2_K through Q8_0 formats with size, speed, and perplexity trade-offs, plus importance matrix calibration for low-bit quantization. - OpenAI-Compatible Server: Deploy llama-server with chat completions, streaming, continuous batching, Docker packaging, and NGINX load balancing. - Use Case: Deploy a Llama 2-7B chatbot on an M3 MacBook by downloading a Q4_K_M GGUF model, building llama.cpp with Metal support, and serving an OpenAI-compatible API on port 8080. ## Quick Start Use the llama-cpp skill to download a Q4_K_M GGUF model from HuggingFace and start an OpenAI-compatible inference server on my local machine.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run Llama models on a Mac without a GPU?▼

Build llama.cpp with Metal support using make LLAMA_METAL=1, download a GGUF model like Llama-2-7B Q4_K_M, and run llama-cli with -ngl 999 to offload layers to the Apple Silicon GPU. An M3 Max achieves roughly 50 tokens per second.

Which GGUF quantization format should I use?▼

Q4_K_M is the recommended default, balancing a 4.1 GB size for 7B models with only about 1.7% perplexity increase over FP16. Use Q6_K or Q8_0 for maximum quality, and Q3_K_M or Q4_K_S for very large models like 70B that must fit in limited memory.

llama.cpp vs vLLM vs TensorRT-LLM for inference?▼

llama.cpp targets CPUs, Apple Silicon, and non-NVIDIA hardware with minimal dependencies. Choose vLLM when you have NVIDIA GPUs and want PagedAttention with a Python API, or TensorRT-LLM for maximum datacenter throughput on A100/H100 hardware.

Can llama.cpp run on AMD or Intel GPUs?▼

Yes, llama.cpp supports AMD GPUs through ROCm by building with make LLAMA_HIP=1, then offloading layers with the -ngl flag. Intel GPUs and CPU-only machines are also supported, making it suitable when CUDA is unavailable.

Why is my llama.cpp inference slow or out of memory?▼

Out-of-memory errors mean the quantization is too large or too many layers are offloaded to GPU; reduce -ngl or use a lower format like Q4_K_S. Slow inference often comes from higher-bit quantization, hyperthreading, or missing BLAS acceleration.

How do I convert a HuggingFace model to GGUF?▼

Download the HuggingFace model, run convert_hf_to_gguf.py to produce an FP16 GGUF file, then use llama-quantize to create your target format such as Q4_K_M. Pre-quantized models are also available from TheBloke on HuggingFace.

Related Skills

Explore More Software Engineering Skills→
lenadlmlenadlm

gguf-quantization

Convert HuggingFace models to GGUF format and quantize them for CPU inference.

Community
Advanced
diskd-aidiskd-ai

cerebras-api

Integrate Cerebras Chat Completions API into Python and TypeScript applications.

Official
Advanced
agentic-inagentic-in
577

Inference

Select inference paths, serving stacks, and benchmarking plans for ML models.

Official
Intermediate
NousResearchNousResearch
239.1k

serving-llms-vllm

Deploy high-throughput LLM inference servers with vLLM, PagedAttention, and quantization.

Official
Advanced