skills.rest
Ryanzucchi
RyanzucchiCOMMUNITY
Explore all skills by this author →
GitHub

otimizacao-de-modelos-generativos-locais

Configures llama.cpp inference with Q4_K_M quantization, KV caching, and asynchronous generation for local language models.

Updated Jul 19, 2026
Search Tags:#latency#quantization#gguf#llama.cpp#inference optimization#local llm#kv cache
One-click install
npx skills add https://github.com/Ryanzucchi/Eldritch_Lich --skill otimizacao-de-modelos-generativos-locais-ryanzucchi
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: otimizacao-de-modelos-generativos-locais
Source: https://github.com/Ryanzucchi/Eldritch_Lich/tree/main/.agents/skills/otimizacao-de-modelos-generativos-locais
Command: npx skills add https://github.com/Ryanzucchi/Eldritch_Lich --skill otimizacao-de-modelos-generativos-locais-ryanzucchi

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Running generative language models locally on a writer's own hardware often causes high VRAM consumption and slow time-to-first-token, making AI writing assistance unusable. This Skill defines how to optimize local inference engines so suggestions arrive fast without exhausting GPU memory. ## Core Features & Use Cases - INT4 Quantization (Q4_K_M): Selects GGUF quantized model variants that keep VRAM usage under 4.5 GB, fitting common 6 GB GPUs like the RTX 3060. - KV Prompt Caching: Configures llama.cpp with --cache-type q8_0 and an 8192-token context so worldbuilding and character sheets stay cached between generation calls, cutting TTFT below 300ms. - Literary Decoding Parameters: Sets temperature 0.80, top_p 0.90, and repeat_penalty 1.10 for creative text quality. - Asynchronous UI Rendering: Runs generation in a Web Worker and displays streaming text at reduced opacity until completion. - Use Case: When a user activates a new local model in the AI assistant settings, apply this Skill to launch llama-server with the correct quantization, cache, and sampling flags so inline suggestions appear in under 300ms. ## Quick Start Configure the local llama.cpp inference engine with Q4_K_M quantization, KV cache enabled, and literary decoding parameters so writing suggestions generate with low latency.

Frequently Asked Questions about otimizacao-de-modelos-generativos-locais

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I reduce VRAM usage when running llama.cpp locally?▼

Use a Q4_K_M quantized GGUF model variant, which keeps the VRAM footprint under 4.5 GB for a 7B-parameter model. This fits consumer GPUs with 6 GB of memory such as the RTX 3060 or RTX 4060.

How to speed up time-to-first-token in llama.cpp inference?▼

Enable KV prompt caching with the --cache-type q8_0 flag and keep the system context (character sheets, worldbuilding) as a stable prompt prefix. With a warm cache, TTFT drops to around 180-300ms instead of over 10 seconds.

What sampling parameters work best for creative writing with local LLMs?▼

Use temperature 0.80 for controlled diversity, top_p 0.90 for vocabulary filtering, and repeat_penalty 1.10 to avoid repeated phrases. Temperatures below 0.5 tend to produce flat, repetitive literary text.

Why does local LLM generation take over 10 seconds for the first token?▼

This happens when KV caching is disabled, forcing the model to reprocess the entire attention context on every call. Enable KV cache at engine startup and keep the context prefix stable across consecutive requests.

When should I not use local inference optimization techniques?▼

Do not apply these techniques to cloud-hosted models, which require server-side optimization instead. They also do not apply to fine-tuning or training, which are offline operations distinct from inference optimization.

Related Skills

Explore More Software Engineering Skills→
attentiondotnetattentiondotnet

gguf-quantization

Quantize model weights to GGUF format using llama.cpp tooling.

Community
Intermediate
syncfusionsyncfusion

syncfusion-javascript-inline-ai-assist

Provide contextual AI text editing with Syncfusion InlineAIAssist popups.

Official
Advanced
girijashankarjgirijashankarj
28

setup-monitoring

Define metrics, thresholds, alerts, and dashboard panels for service monitoring.

Community
Intermediate
iani-kuliiani-kuli

llama-cpp

Discover and run quantized GGUF models locally via llama.cpp and Hugging Face.

Community
Advanced