skills.rest
kaminocorp
kaminocorpOFFICIAL
Explore all skills by this author →
Website𝕏@kaminocorpoGitHub

llama-cpp

Runs GGUF-quantized LLM inference on CPUs, Apple Silicon, and non-NVIDIA GPUs.

1|Updated Mar 12, 2026
Search Tags:#apple silicon#quantization#edge deployment#gguf#llama.cpp#cpu inference#llm inference
One-click install
npx skills add https://github.com/kaminocorp/hermes-alpha-hunter --skill llama-cpp-kaminocorp
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/kaminocorp/hermes-alpha-hunter/tree/main/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/kaminocorp/hermes-alpha-hunter --skill llama-cpp-kaminocorp

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python, and includes references (resource) components.

What problem does it solve? Running large language models typically requires expensive NVIDIA GPUs and CUDA infrastructure, leaving CPU-only machines, Apple Silicon Macs, AMD/Intel GPUs, and edge devices without a viable inference path. This Skill provides guidance for deploying LLMs on commodity hardware using llama.cpp with GGUF quantization. ## Core Features & Use Cases - Cross-platform inference: Run models on CPU, Apple Silicon (Metal), AMD (ROCm), Intel GPUs, and embedded devices like Raspberry Pi. - GGUF quantization: Reduce model memory footprint with 1.5-8 bit quantization formats (Q4_K_M recommended) for 4-10x speedup over PyTorch on CPU. - OpenAI-compatible server: Deploy llama-server with chat completions, streaming, continuous batching, and Docker support. - Use Case: Deploy a Llama 2-7B chatbot on an M3 MacBook by downloading a Q4_K_M GGUF model from HuggingFace and serving it via llama-server at 40-60 tokens per second with zero cloud cost. ## Quick Start Use the llama-cpp skill to download a Q4_K_M GGUF model from HuggingFace and start an OpenAI-compatible inference server on port 8080.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run LLM inference on a CPU without a GPU?▼

Use llama.cpp with a GGUF-quantized model such as Q4_K_M, which runs a 7B model at 30-50 tokens per second on modern CPUs. Build from source with make, download a model from HuggingFace, and run llama-cli with the model path.

What GGUF quantization format should I use?▼

Q4_K_M is the recommended default, offering 4.1 GB size for 7B models with only 1.7% perplexity increase over FP16. Use Q6_K or Q8_0 for maximum quality, or Q3_K_M for very large models like 70B that must fit in limited RAM.

llama.cpp vs vLLM vs TensorRT-LLM: which should I choose?▼

Choose llama.cpp for CPU, Apple Silicon, AMD/Intel GPUs, or edge deployment. Use vLLM or TensorRT-LLM when you have NVIDIA GPUs and need maximum datacenter throughput with features like PagedAttention.

Does llama.cpp work on Apple Silicon Macs?▼

Yes, llama.cpp supports Apple Silicon via Metal acceleration. Build with make LLAMA_METAL=1 and offload layers with -ngl 999; an M3 Max achieves 40-60 tokens per second on Llama 2-7B Q4_K_M.

Why does my quantized model output gibberish?▼

Gibberish output usually means the quantization is too aggressive, such as Q2_K which degrades perplexity by over 15%. Switch to Q4_K_M or Q5_K_M, verify the model converted correctly, and consider using an importance matrix for low-bit formats.

How do I deploy llama.cpp as an API server?▼

Run llama-server with your GGUF model to get an OpenAI-compatible /v1/chat/completions endpoint with streaming support. It can be containerized with Docker, scaled behind NGINX load balancing, and monitored via built-in metrics endpoints.

Related Skills

Explore More Software Engineering Skills→
lenadlmlenadlm

gguf-quantization

Convert HuggingFace models to GGUF format and quantize them for CPU inference.

Community
Advanced
FortiumPartnersFortiumPartners

using-openai-platform

Integrate OpenAI GPT-5 models, Chat Completions, Responses API, embeddings, and tool calling into applications.

Official
Intermediate
agentic-inagentic-in
577

Inference

Select inference paths, serving stacks, and benchmarking plans for ML models.

Official
Intermediate
GregsGreyCodeGregsGreyCode
6

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT.

Community
Advanced