skills.rest
tuliang1024
tuliang1024COMMUNITY
Explore all skills by this author →
GitHub

kvcache-optimization

Optimize KVCache handling for LLM inference to reduce memory usage and improve throughput.

Updated Apr 11, 2026
Search Tags:#optimization#llm-inference#npu#mla#paged-attention#kvcache#fusion-ops
One-click install
npx skills add https://github.com/tuliang1024/cann-recipes-infer --skill kvcache-optimization
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: kvcache-optimization
Source: https://github.com/tuliang1024/cann-recipes-infer/tree/main/.agent/skills/kvcache-optimization
Command: npx skills add https://github.com/tuliang1024/cann-recipes-infer --skill kvcache-optimization

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

KVCache optimization for LLM/ multimodal inference to reduce memory usage and improve throughput, by selecting and combining cache strategies such as continuous caching, paging attention with FA fusion operators, and MLA-based compression. It outlines trigger scenarios including KVCache management, paging attention, KV compression, FA fusion operators, OOM/performance issues, and block_table/slot_mapping construction, offering model-type and scenario-based recommendations.

Core Features & Use Cases

  • Guidance on choosing KVCache modes for standard LLMs, high-performance LLMs, MLA architectures, and diffusion/video models.
  • Detailed implementation patterns: mode1 continuous caching, mode2 paging attention with block_table/slot_mapping, mode3 MLA cache compression, and how to compose them.
  • Real-world use case examples showing how to deploy KVCache strategies to reduce memory footprint while maintaining throughput.

Quick Start

Execute the KVCache optimization by selecting a mode (continuous cache, paging attention with FA, or MLA compression) and implement the corresponding KV cache storage, block_table mapping, and fused attention flow for your model.

Frequently Asked Questions about kvcache-optimization

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize KVCache to reduce memory usage and improve LLM inference throughput?▼

Optimize KVCache by selecting and combining strategies like continuous caching, paging attention with FA fusion operators, and MLA-based compression to reduce memory footprint and improve LLM inference throughput.

What is the best way to implement paged attention and block_table mapping for LLM inference?▼

The best way to implement paged attention is using mode2, which involves constructing block_table and slot_mapping alongside FA fusion operators to manage memory efficiently during LLM inference.

Does MLA cache compression work for both standard LLMs and high-performance architectures?▼

MLA cache compression, deployed as mode3, works across standard LLMs, high-performance models, and MLA architectures, providing specific recommendations based on model type and scenario constraints.

When do I need to use KV compression and FA fusion operators for model inference?▼

You need KV compression and FA fusion operators when encountering OOM or performance issues during LLM inference, triggering scenario-based recommendations for memory management and throughput improvement.

How do I know which KVCache mode to choose for my specific model type?▼

Choose your KVCache mode by evaluating your model type and constraints, selecting from continuous caching, paging attention with FA, or MLA compression based on scenario-specific recommendations.

Why does my LLM inference run out of memory when handling large KVCache?▼

LLM inference runs out of memory with large KVCache due to inefficient storage strategies; applying paging attention with block_table construction or MLA compression reduces the memory footprint.

Related Skills

Explore More Software Engineering Skills→
HTRTanHTRTan

deepseek-api

Selects DeepSeek models and configures inference parameters for API integration.

Community
Intermediate
superfhpsuperfhp

optimizing-attention-flash

Optimize transformer attention with Flash Attention for long sequences.

Community
Advanced
qcmuuqcmuu

serving-llms-vllm

Deploy vLLM inference endpoints with quantization and tensor parallelism.

Community
Advanced
kettleofketchupkettleofketchup

context-engineering

Document patterns for AI agent context management across architectures.

Community
Advanced