kvcache-fa-precision-debug

Debug Flash Attention precision issues by systematically comparing baseline and optimized outputs, isolating Prefill and Decode misalignment, and validating FA invocation parameters like atten_mask and actual_seq_lengths to locate root causes.

NaN or Inf values during inference often stem from KVCache writes or FA fusion operator discrepancies; diagnose by checking block_table configurations, kv_len, and sparse_mode settings against baseline tensor comparisons.

Fix Prefill and Decode precision inconsistencies by performing module-level isolation to compare FA computations and post-processing steps, ensuring actual_seq_lengths and kv_len parameters align across both inference phases.

Yes, KVCache precision debugging supports both single-device and multi-device setups, validating atten_mask, block_table, and actual_seq_lengths across distributed configurations to resolve discrepancies between baseline and optimized outputs.

Exaggerated precision changes under quantization typically arise from FA fusion operator mismatches; diagnose by validating FA invocation parameters and comparing quantized tensor outputs against baseline to isolate the affected modules.

Validate FA invocation parameters including atten_mask, sparse_mode, actual_seq_lengths, kv_len, and block_table, then compare optimized outputs against baseline tensors to systematically isolate the precision failure root cause.