model-pruning

Use Wanda or SparseGPT one-shot pruning with a small calibration dataset of about 128 samples. Wanda computes importance as weight magnitude times input activation norm, then removes the lowest-importance weights to reach your target sparsity in minutes.

Wanda is simpler and faster with O(n) complexity per layer, losing about 0.8% accuracy at 50% sparsity. SparseGPT uses second-order Hessian information for slightly better quality (0.4% loss) but is slower and more memory-intensive. Use Wanda unless you need the absolute best quality.

Unstructured pruning reduces memory by 50% but gives no speedup due to irregular sparsity. N:M structured pruning (2:4 or 4:8) achieves 2× speedup on NVIDIA GPUs with sparse tensor cores, such as the A100.

30% sparsity is conservative with under 0.5% accuracy loss, while 50% is the recommended balance at roughly 1% loss. Beyond 70% sparsity, expect 2-5% degradation, and 90% causes significant quality loss unless done iteratively with fine-tuning.

Removing too many weights at once destroys model capacity without recovery. Use gradual magnitude pruning or iterative prune-and-finetune cycles, increasing sparsity in steps with short fine-tuning runs between iterations to recover accuracy.