Run GGUF models locally with llama.cpp and locate quantized files on the Hugging Face Hub.
npx skills add https://github.com/infinition/LaRuche --skill llama-cpp-infinition
Please help me install this Agent Skill. Skill: llama-cpp Source: https://github.com/infinition/LaRuche/tree/main/laruche/skills/llama-cpp Command: npx skills add https://github.com/infinition/LaRuche --skill llama-cpp-infinition
apps=llama.cpp filter, read the local-app page for the maintainer's recommended quant, and confirm exact filenames and sizes via the tree API before downloading. - Quantization Guidance: Match quant levels (Q4_K_M, Q5_K_M, IQ variants, Q8_0) to hardware constraints using file size, context memory cost, and headroom arithmetic. - Serving and Python Integration: Launch an OpenAI-compatible llama-server, or use llama-cpp-python with GPU offload via n_gpu_layers, including CUDA and Metal builds. - Use Case: A user with 16 GB of RAM wants to run a coding model locally. The skill finds a verified GGUF repo, recommends Q5_K_M with its exact size, and produces a working llama-server -hf <repo>:<QUANT> -c 4096 command. ## Quick Start Ask the assistant to find a GGUF model for your hardware on Hugging Face and give you the exact llama-server command to run it locally.High-intent search queries and answers about installing and using this skill.
Install llama.cpp via brew or winget, then run llama-server with the -hf flag pointing to a Hugging Face repo and quant label, such as llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q4_K_M -c 4096. Verify the installation first with llama-cli --version.
Search huggingface.co/models with the apps=llama.cpp filter to list only repos containing GGUF files. Then confirm exact filenames and byte sizes through the tree API at /api/models/<repo>/tree/main?recursive=true, since model cards can list quants that were never uploaded.
Q4_K_M is the default balance of quality and size for general chat. Use Q5_K_M or Q6_K for code if memory allows, Q3_K_M or IQ variants when space is tight, and Q8_0 when memory is unconstrained. The file size is the floor—add context memory and headroom on top.
Yes, but GPU support is compiled in at install time. Reinstall with CMAKE_ARGS="-DGGML_CUDA=on" or "-DGGML_METAL=on" plus --force-reinstall --no-cache-dir, otherwise pip serves the cached CPU wheel. Set n_gpu_layers to control how many layers move to the GPU.
The model file fitting in RAM is not enough—context size (-c) consumes additional memory that grows during conversation. Lower -c first since it is free to change, then drop to a smaller quantization if the process still dies.
mmproj-*.gguf is the vision projector for multimodal models, loaded alongside the main checkpoint. It produces nothing on its own and must not be confused with the quantized language model files.
AMGrobelnikDetect CPUs, RAM, GPU/VRAM, and disk space, including cgroup limits.
jayll1303Run local GGUF model inference and OpenAI-compatible serving with llama.cpp.
AevonixServe LLM inferences with vLLM using PagedAttention and continuous batching.
big4council-progDeploy high-throughput LLM inference endpoints with vLLM and OpenAI-compatible APIs.