skills.rest
azaanaliraza
azaanalirazaCOMMUNITY
Explore all skills by this author →
GitHub

llama-cpp

Run and quantize GGUF LLMs with llama.cpp across CPU, Apple Silicon, and GPU hardware.

Updated Apr 18, 2026
Search Tags:#apple silicon#quantization#edge deployment#gguf#llama.cpp#local llm#llm inference
One-click install
npx skills add https://github.com/azaanaliraza/operarius --skill llama-cpp-azaanaliraza
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/azaanaliraza/operarius/tree/main/src-tauri/bin/hermes/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/azaanaliraza/operarius --skill llama-cpp-azaanaliraza

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python>=0.2.0, and includes references (resource) components.

What problem does it solve? Running large language models locally without NVIDIA GPUs or cloud APIs is difficult, and converting HuggingFace models into efficient quantized formats requires deep knowledge of tooling. This Skill provides complete workflows for GGUF conversion, quantization, and inference with llama.cpp on any hardware. ## Core Features & Use Cases - GGUF Conversion & Quantization: Convert HuggingFace models to GGUF and quantize with K-quants (Q2_K through Q8_0) plus imatrix calibration for better low-bit quality. - Cross-Platform Inference: Run models on CPU, Apple Silicon (Metal), NVIDIA (CUDA), and AMD (ROCm) via CLI, Python bindings, or an OpenAI-compatible server. - Ecosystem Integration: Deploy through Ollama, LM Studio, text-generation-webui, or connect OpenAI clients to llama-server. - Use Case: Convert Llama-3.1-8B to Q4_K_M GGUF with an importance matrix, then serve it on an M3 Mac via llama-server so your app can call a local OpenAI-compatible chat endpoint. ## Quick Start Convert my HuggingFace model to a Q4_K_M GGUF file and start a local OpenAI-compatible server on port 8080 with GPU offloading.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I convert a HuggingFace model to GGUF format?▼

Use the convert_hf_to_gguf.py script from llama.cpp to convert the model to FP16 GGUF, then run llama-quantize to produce a quantized file like Q4_K_M. Pre-quantized GGUF models are also available from TheBloke on HuggingFace.

What GGUF quantization level should I use for a 7B model?▼

Q4_K_M is the recommended default, offering about 4.1 GB size with only ~1.7% perplexity increase over FP16. Use Q5_K_M or Q6_K for higher quality, or Q3_K_M when memory is severely constrained.

Does llama.cpp work on Apple Silicon Macs?▼

Yes, llama.cpp supports Apple Silicon through Metal acceleration. Build with GGML_METAL=1 and set n_gpu_layers to 99 to offload all layers, achieving roughly 40-60 tokens per second on an M3 Max with a 7B Q4_K_M model.

How do I run llama.cpp as an OpenAI-compatible API server?▼

Launch llama-server with your GGUF model, host, port, and GPU layer flags, then point any OpenAI client at http://localhost:8080/v1. It supports chat completions, streaming, continuous batching, and parallel request slots.

Why does my quantized model produce garbage output?▼

Garbage output usually comes from overly aggressive quantization like Q2_K, a wrong chat_format setting, or a corrupted model file. Test the FP16 baseline first, use imatrix calibration for Q4 and below, and verify with low temperature.

When should I use vLLM instead of llama.cpp?▼

Choose vLLM when you have NVIDIA GPUs and need maximum throughput with PagedAttention in a Python-first stack. llama.cpp is better for CPU-only machines, Apple Silicon, AMD/Intel GPUs, and edge deployment without Python.

Related Skills

Explore More Software Engineering Skills→
math-incmath-inc
1.2k

gguf-quantization

Convert HuggingFace models to GGUF and quantize for CPU or GPU inference.

Official
Advanced
automatedigitalautomatedigital
1

llama-cpp

Run GGUF quantized LLM inference on CPUs, Apple Silicon, and AMD/Intel GPUs.

Official
Intermediate
cyberpunk042cyberpunk042

infra-cache

Configure per-model prompt and KV cache settings in LocalAI YAML files.

Community
Intermediate
richardnguyen0715richardnguyen0715

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for high throughput and low latency.

Community
Advanced