skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

optimizing-attention-flash

Optimizes transformer attention with Flash Attention for faster training and lower memory usage.

1|Updated Mar 5, 2026
Search Tags:#pytorch#memory efficiency#transformers#fp8#flash attention#long context#gpu optimization
One-click install
npx skills add https://github.com/Clay-HHK/claude-skills --skill optimizing-attention-flash-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: optimizing-attention-flash
Source: https://github.com/Clay-HHK/claude-skills/tree/main/flash-attention
Command: npx skills add https://github.com/Clay-HHK/claude-skills --skill optimizing-attention-flash-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires flash-attn, torch, transformers, and includes references (resource) components.

What problem does it solve? Standard transformer attention consumes quadratic memory and becomes a bottleneck when training or running inference on long sequences, often causing GPU out-of-memory errors and slow throughput. ## Core Features & Use Cases - PyTorch Native Integration: Enable Flash Attention through torch.nn.functional.scaled_dot_product_attention on PyTorch 2.2+ with no extra dependencies. - flash-attn Library Workflows: Use advanced features like multi-query attention, sliding window attention, and causal masking via the flash-attn package. - H100 FP8 Optimization: Run FlashAttention-3 with FP8 precision on Hopper GPUs for up to 1.2 PFLOPS throughput. - Use Case: When fine-tuning a Llama 2 7B model with 8K context hits OOM on an A100, switch the attention implementation to Flash Attention 2 to cut memory by over 50% and gain roughly 3x training speedup. ## Quick Start Enable Flash Attention in my PyTorch transformer model and verify the speedup with a benchmark.

Frequently Asked Questions about optimizing-attention-flash

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I enable Flash Attention in PyTorch?▼

Use torch.nn.functional.scaled_dot_product_attention with PyTorch 2.2 or later, which automatically dispatches to Flash Attention kernels when available. You can force the backend with torch.backends.cuda.sdp_kernel(enable_flash=True).

How do I use Flash Attention with HuggingFace Transformers?▼

Pass attn_implementation="flash_attention_2" to AutoModel.from_pretrained along with torch_dtype=torch.float16. Transformers 4.36+ supports this for Llama, Mistral, Falcon, Qwen, Gemma, and other architectures.

Flash Attention vs xFormers: which should I use?▼

Flash Attention is the better choice when you need raw speed and memory reduction for standard attention patterns on GPU. xFormers offers a wider variety of attention variants, so use it when you need specialized attention types beyond speed optimization.

Does Flash Attention work on V100 or older GPUs?▼

No, Flash Attention requires NVIDIA Turing (compute capability 7.5) or newer, with full support on Ampere GPUs like A100 and A10. V100 (Volta) and CPU inference are not supported.

Why is Flash Attention not giving me a speedup?▼

Speedup scales with sequence length: sequences under 512 tokens see only 10-20% improvement, while 2K+ tokens see 3-4x. Also verify inputs are float16 or bfloat16, since float32 is not supported by Flash Attention kernels.

How do I fix ImportError when installing flash-attn?▼

Install with pip install flash-attn --no-build-isolation, which resolves most build failures. If it still fails, install the CUDA toolkit first via conda install cuda -c nvidia, then retry the installation.

Related Skills

Explore More Software Engineering Skills→
AlvaroBianoAlvaroBiano

optimizing-attention-flash

Optimizes transformer attention for memory-efficient training and inference on long sequences.

Community
Advanced
flagos-aiflagos-ai
18

perf-test-flagos

Benchmark vLLM-served AI models across five workload profiles for throughput, latency, TTFT, and TPOT.

Official
Advanced
Mathews-TomMathews-Tom
310

gpu-optimizer

Optimize PyTorch and CUDA training on NVIDIA GPUs with 8–24GB VRAM.

Community
Advanced
brittaniebuffiecsubrittaniebuffiecsu
1

optimizing-attention-flash

Optimize PyTorch transformer attention with Flash Attention for long sequences.

Community
Advanced