skills.rest
waybarrios
waybarriosCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

huggingface-local-models

Selects and runs GGUF models from Hugging Face locally with llama.cpp.

507|40|Updated Apr 26, 2026
Search Tags:#quantization#gguf#llama.cpp#local llm#model inference#hugging face#llama-server
One-click install
npx skills add https://github.com/waybarrios/opencode-power-pack --skill huggingface-local-models-waybarrios
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: huggingface-local-models
Source: https://github.com/waybarrios/opencode-power-pack/tree/main/skills/huggingface-local-models
Command: npx skills add https://github.com/waybarrios/opencode-power-pack --skill huggingface-local-models-waybarrios

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Choosing the right quantized model and running it locally is confusing: GGUF repos contain many quant variants, hardware constraints differ across CPU, Metal, CUDA, and ROCm, and exact filenames are hard to confirm. This Skill provides a URL-first workflow to find llama.cpp-compatible models on Hugging Face, pick the right quantization, and launch them with llama-cli or llama-server. ## Core Features & Use Cases - Hub-first model discovery: Search Hugging Face with the llama.cpp app filter, read the local-app page for the recommended quant, and confirm exact .gguf filenames via the tree API. - Quantization guidance: Detailed format comparison tables (Q4_K_M, Q5_K_M, Q6_K, Q8_0, and more), size scaling per model parameter count, and imatrix calibration workflows. - Hardware acceleration setup: Build and run instructions for Apple Metal, NVIDIA CUDA, AMD ROCm, and CPU with BLAS. - Use Case: You want to run a 35B model on a Mac with limited RAM. Use this Skill to find the repo, confirm the UD-Q4_K_M file fits your memory budget, and launch an OpenAI-compatible local server with llama-server. ## Quick Start Ask the assistant to find a llama.cpp-compatible GGUF model on Hugging Face for your hardware and start it as a local OpenAI-compatible server.

Frequently Asked Questions about huggingface-local-models

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a Hugging Face model locally with llama.cpp?▼

Use llama-cli or llama-server with the -hf flag pointing to a GGUF repo and quant, for example llama-server -hf repo:Q4_K_M. For repos with custom naming, use --hf-repo plus --hf-file with the exact filename from the tree API.

Which GGUF quantization should I choose for local inference?▼

Q4_K_M is the recommended default for the best balance of quality and speed. Prefer Q5_K_M or Q6_K for code or technical workloads when memory allows, and Q3_K_M or IQ variants for tighter RAM or VRAM budgets.

How do I find the exact GGUF filename in a Hugging Face repo?▼

Query the tree API at https://huggingface.co/api/models/<repo>/tree/main?recursive=true and filter entries where type is file and path ends with .gguf. The path field gives the exact filename to pass to --hf-file.

Does llama.cpp support GPU acceleration on Mac and NVIDIA?▼

Yes. Build with GGML_METAL=1 for Apple Silicon or GGML_CUDA=1 for NVIDIA, then offload layers with -ngl. AMD GPUs use ROCm via LLAMA_HIP=1, and CPU-only runs can use OpenBLAS acceleration.

When should I convert a model to GGUF myself?▼

Convert only when the repo does not already publish GGUF files. Download the Transformers weights, run convert_hf_to_gguf.py to produce an f16 GGUF, then quantize with llama-quantize to your target format.

Why does my local model output gibberish after quantization?▼

Gibberish usually means the quantization is too aggressive, such as Q2_K. Switch to Q4_K_M or Q5_K_M, verify the conversion completed correctly, and consider using an importance matrix (imatrix) for low-bit quants.

Related Skills

Explore More Software Engineering Skills→
big4council-progbig4council-prog

llama-cpp

Select llama.cpp GGUF quantizations and launch commands from Hugging Face repositories.

Community
Advanced
copilot-ldcopilot-ld
1

libllm

Expose an HTTP LLM API client for OpenAI-compatible chat completions and embeddings.

Official
Advanced
flagos-aiflagos-ai
18

perf-test-flagos

Benchmark vLLM-served AI models across five workload profiles for throughput, latency, TTFT, and TPOT.

Official
Advanced
cuba6112cuba6112

unsloth-gguf

Export fine-tuned machine learning models to GGUF format with quantization and LoRA merging.

Community
Intermediate