skills.rest
superfhp
superfhpCOMMUNITY
Explore all skills by this author →
GitHub

llama-cpp

Runs local GGUF model inference and discovers quantized models on Hugging Face Hub.

Updated May 13, 2026
Search Tags:#quantization#edge deployment#gguf#llama.cpp#local inference#hugging face#llama-server
One-click install
npx skills add https://github.com/superfhp/lumi-agent-body --skill llama-cpp-superfhp
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/superfhp/lumi-agent-body/tree/main/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/superfhp/lumi-agent-body --skill llama-cpp-superfhp

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python, and includes references (resource) components.

What problem does it solve? Running large language models locally requires choosing the right GGUF quantization, finding compatible model files on Hugging Face, and configuring llama.cpp for your specific hardware, which involves many manual steps and platform-specific pitfalls. ## Core Features & Use Cases - Hub-First Model Discovery: Search Hugging Face for llama.cpp-compatible repos, read the local-app page for recommended quants, and confirm exact GGUF filenames and sizes via the tree API. - Local Inference on Any Hardware: Run models on CPU, Apple Silicon, CUDA, ROCm, or Intel GPUs using llama-cli, llama-server, or the llama-cpp-python bindings with chat, streaming, and embedding support. - Quantization Guidance: Choose between Q4_K_M, Q5_K_M, Q6_K, IQ variants, and repo-specific labels based on RAM/VRAM budgets and quality requirements. - Use Case: A developer wants to run a 24B-parameter model on a laptop with 16 GB of VRAM. The skill finds the right GGUF repo on Hugging Face, identifies the UD-Q4_K_M quant that fits, and produces the exact llama-server command to launch an OpenAI-compatible endpoint. ## Quick Start Ask the assistant to find a GGUF version of a Hugging Face model and give you the llama-server command to run it locally.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a GGUF model locally with llama.cpp?▼

Install llama.cpp via brew, winget, or a CMake build, then run llama-cli or llama-server with the -hf flag pointing to a Hugging Face repo and quant, such as llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0. You can also load a local .gguf file directly with the -m flag.

How do I find GGUF files for a Hugging Face model?▼

Open the repo with ?local-app=llama.cpp to see the recommended quant and launch snippet, then query the tree API at /api/models/<repo>/tree/main?recursive=true to list exact .gguf filenames and byte sizes. Separate main checkpoints from mmproj projector files and BF16 shards.

Which GGUF quantization should I use for my hardware?▼

Start with Q4_K_M for general chat, Q5_K_M or Q6_K for code and technical work, and Q3 or IQ variants only for tight memory budgets. Prefer the exact quant label shown in the Hugging Face hardware compatibility section over generic tables.

Does llama-cpp-python support GPU acceleration?▼

Yes, install with CMAKE_ARGS="-DGGML_CUDA=on" for NVIDIA GPUs or "-DGGML_METAL=on" for Apple Silicon, then set n_gpu_layers to offload layers to the GPU. Use 0 for CPU-only or a high value like 99 to offload everything.

Why does my GGUF model output gibberish after quantization?▼

Gibberish usually means the quantization is too aggressive, such as Q2_K, or the model converted incorrectly. Try Q4_K_M or Q5_K_M, use an importance matrix for calibration, and verify the FP16 version produces sensible output first.

Can llama-server provide an OpenAI-compatible API?▼

Yes, llama-server exposes /v1/chat/completions with streaming support, so OpenAI client libraries work by setting base_url to http://localhost:8080/v1. It also provides /metrics and /health endpoints for monitoring.

Related Skills

Explore More Software Engineering Skills→
RachasumanthRachasumanth

model-publisher

Package trained AI models and publish them to Hugging Face Hub.

Community
Advanced
sylin-orgsylin-org
4

koan-ai-integration

Integrate chat endpoints, embeddings, and RAG workflows with vector search.

Official
Advanced
tadod12tadod12

serving-llms-vllm

Serve LLM inference via OpenAI-compatible endpoints using vLLM.

Community
Advanced
tdiminotdimino
41

llama-cpp

Run GGUF model inference with llama.cpp and apply LoRA adapters.

Community
Advanced