skills.rest
shakfu
shakfuCOMMUNITY
Explore all skills by this author →
GitHub

debug-model

Diagnose silent output corruption in MAX model ports using per-layer tensor-dump comparison against a reference.

Updated Sep 15, 2026
Search Tags:#model debugging#cosine similarity#parity#max#tensor comparison#llm porting#greedy divergence
One-click install
npx skills add https://github.com/shakfu/mdsp --skill debug-model-shakfu
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: debug-model
Source: https://github.com/shakfu/mdsp/tree/main/.claude/skills/debug-model
Command: npx skills add https://github.com/shakfu/mdsp --skill debug-model-shakfu

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires numpy, torch, transformers, and includes references (resource) components.

What problem does it solve? A MAX model that loads, compiles, serves, and generates tokens can still produce wrong output — gibberish, greedy divergence, or logits that match but argmax that differs. Scalar print taps and blind recompile loops hide directional bugs and burn GPU time. This Skill provides a disciplined protocol that localizes silent corruption numerically before any fix is attempted. ## Core Features & Use Cases - Per-layer tensor comparator: Build HF and MAX dumpers that save FP32 hidden states at every layer, then compare cosine similarity, mean/max diff, and spike positions offline. - Signature-based diagnosis: Interpret comparator output patterns (position-dependent RoPE bugs, false cliffs from bad dump indexing, attention-sink failures) using the token-0 invariant and per-token/per-dim cosine slices. - Serve-vs-pipeline bisect: When teacher-forced dumps match but generated text diverges, isolate decode-state (KV/conv cache) bugs from harness bugs (tokenizer, chat template) without touching the graph. - Use Case: During a custom-architecture port, greedy output diverges at token 3. Build the comparator, find a t≥1 cliff from layer 1, check RoPE inv_freq, discover HF uses NoPE at runtime, fix the graph, and verify with a full comparator pass. ## Quick Start Ask the agent to debug why my MAX model port serves but generates text that diverges from the Hugging Face reference, starting with a per-layer tensor-dump comparison.

Frequently Asked Questions about debug-model

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I debug a MAX model that serves but generates wrong tokens?▼

Build a per-layer tensor-dump comparator: dump FP32 hidden states from both the Hugging Face reference and the MAX graph on the same prompt, then compare cosine similarity per layer. The first diverging layer localizes the bug before any recompile.

Why do logits match but greedy tokens still diverge in my model port?▼

High logit cosine with wrong argmax indicates near-miss logits from rank-1 subspace drift, or a harness bug rather than a graph bug. Compare top-10 logit diffs, then bisect teacher-forced dumps versus incremental pipeline decode versus serve output.

What causes a cosine similarity cliff at one decoder layer?▼

A cliff at layer N with HF spiking and MAX flat usually means attention failed to form an anchor, while MAX spiking suggests wrong norm order or a dump bug. First rule out false cliffs from wrong hidden_states indexing or a missing attention_mask on decode-prefix dumps.

How do I check if RoPE is causing position-dependent divergence?▼

If token 0 matches perfectly but tokens t≥1 diverge from layer 1, inspect the HF rotary embedding: print inv_freq, cos, and sin values. Near-zero inv_freq means HF uses identity rotation at runtime, so the MAX graph must not apply plain RoPE.

When should I not use tensor-dump parity debugging?▼

Skip this protocol when the server crashes on load or the graph implementation is unfinished — those are import and scaffolding problems. It also does not apply to tuning logit-comparison tolerances on an already-verified model.

Why does verification still fail after fixing one graph bug?▼

Silent corruption often involves stacked independent bugs: comparator harness bugs, graph math bugs, pipeline decode-state bugs, and serve harness bugs. Fixing one moves the failure index forward, so run full end-to-end verification after each fix rather than re-dumping at the same index.

Related Skills

Explore More Software Engineering Skills→
amdamd
154

quark-torch-model-intake

Analyze HuggingFace model architectures for Quark PTQ planning readiness.

Official
Advanced
Mesh-LLMMesh-LLM
3.4k

benchmark-tune

Runs mesh-llm benchmark tune sweeps to optimize model-serving throughput settings.

Official
Intermediate
edwinhuedwinhu
19

notebook-debug

Inspect .ipynb files for tracebacks and cell execution issues.

Community
Intermediate
MGriotMGriot

git-bisector

Pinpoint the first bug-introducing commit via binary search over git history.

Community
Intermediate