skills.rest
Orchestra-Research
Orchestra-ResearchOFFICIAL
Explore all skills by this author →
GitHub

llama-cpp

Runs quantized GGUF LLM inference on CPUs, Apple Silicon, and non-NVIDIA GPUs.

13.0k|930|Updated Nov 3, 2025
Search Tags:#apple silicon#quantization#edge deployment#gguf#llama.cpp#llm serving#cpu inference
One-click install
npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill llama-cpp-orchestra-research
Or copy as Structured Prompt for Agentâ–¼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/Orchestra-Research/AI-research-SKILLs/tree/main/12-inference-serving/llama-cpp
Command: npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill llama-cpp-orchestra-research

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python, and includes references (resource) components.

What problem does it solve? Running large language models typically requires expensive NVIDIA GPUs and CUDA infrastructure, leaving CPU-only machines, Apple Silicon Macs, AMD/Intel GPUs, and edge devices without a viable inference path. ## Core Features & Use Cases - Cross-Platform Inference: Run LLMs on CPU-only servers, M1/M2/M3/M4 Macs via Metal, AMD GPUs via ROCm, and embedded devices like Raspberry Pi. - GGUF Quantization: Compress models to 1.5-8 bit formats (Q4_K_M, Q6_K, Q8_0) to reduce memory footprint and achieve 4-10x speedup over PyTorch on CPU. - OpenAI-Compatible Server: Deploy llama-server with /v1/chat/completions endpoints, streaming, continuous batching, and Docker support. - Use Case: Deploy a Llama 2-7B chatbot on an M3 MacBook by downloading a Q4_K_M GGUF model and running llama-cli with Metal acceleration for 40-60 tokens per second. ## Quick Start Download a Q4_K_M GGUF model from HuggingFace and run llama-cli with the model path and a prompt to generate a response.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run LLM inference on a Mac without a GPU?â–¼

Build llama.cpp with Metal support using make LLAMA_METAL=1, then run llama-cli with a GGUF model and -ngl 999 to offload all layers. An M3 Max achieves 40-60 tokens per second on Llama 2-7B Q4_K_M.

What GGUF quantization format should I use?â–¼

Q4_K_M is the recommended default, offering 4.1 GB size for 7B models with only 1.68% perplexity increase over FP16. Use Q6_K or Q8_0 for maximum quality, or Q3_K_M for very large models like 70B.

llama.cpp vs vLLM vs TensorRT-LLM for inference?â–¼

Use llama.cpp for CPU, Apple Silicon, AMD/Intel GPUs, or edge devices. Choose vLLM or TensorRT-LLM when you have NVIDIA GPUs and need maximum datacenter throughput with PagedAttention.

Does llama.cpp support AMD and Intel GPUs?â–¼

Yes, build with make LLAMA_HIP=1 for AMD GPUs via ROCm, then offload layers with -ngl. Intel GPUs and CPU-only execution are also supported without CUDA dependencies.

Why is my llama.cpp inference slow or out of memory?â–¼

Out-of-memory errors mean you should reduce GPU offloaded layers (-ngl), use a lower quantization like Q4_K_S, or shrink context with -c 2048. Slow inference often comes from higher-bit quantization or hyperthreading; set -t to physical core count.

How do I deploy llama.cpp as an API server?â–¼

Run llama-server with your GGUF model, --host 0.0.0.0, and --port 8080 to expose OpenAI-compatible /v1/chat/completions endpoints with streaming support. Docker deployment and NGINX load balancing are documented for production use.

Related Skills

Explore More Software Engineering Skills→
lenadlmlenadlm

gguf-quantization

Convert HuggingFace models to GGUF format and quantize them for CPU inference.

Community
Advanced
qcmuuqcmuu

serving-llms-vllm

Deploy vLLM inference endpoints with quantization and tensor parallelism.

Community
Advanced
qcmuuqcmuu

gguf-quantization

Convert HuggingFace checkpoints to GGUF and apply llama.cpp quantization.

Community
Intermediate
zhouboyu-xrealzhouboyu-xreal

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT and quantization.

Community
Advanced