parity-testing

Load both models on CPU with float32, run a state dict round-trip check for exact weight equality, then compare logits from identical seeded inputs with max_diff under 1e-5. Finish with an end-to-end GPU test in bfloat16 using looser tolerances.

Isolate the failing component by testing attention, MLP, norm, and RoPE separately, then verify weight loading via state dict round-trip. If weights match, force baseline backends like SDPA attention and torch linear, and use injection of reference outputs to pinpoint the diverging component.

GPU execution introduces non-determinism from parallel reductions, kernel launch order, and lower-precision dtypes like bfloat16. MoE models additionally suffer from non-deterministic top-k routing, so MoE parity should be tested on CPU or with fixed routing.

For bfloat16 end-to-end tests on GPU, use max_diff below 1e-2 and cosine similarity above 0.9999. For float32 on CPU, require max_diff below 1e-5 at component level and exact equality for state dict round-trips.

When tie_word_embeddings is enabled, lm_head.weight and embed_tokens.weight must share the same underlying storage. If checkpoint loading creates independent copies, parameter counts increase and the weights diverge during training; verify aliasing by comparing data pointers.