research.attention.probe

Activation patching replaces a corrupted prompt's activations with clean-run activations at a specific component using run_with_hooks and fwd_hooks. Measure the logit difference on the target token; strong evidence requires 64+ prompts, 3 seeds, and a shuffled-label control where the effect disappears.

TransformerLens (HookedTransformer) suits models up to 7B parameters with full local intervention support. nnsight with remote tracing is recommended for 8B-70B models to reduce memory usage. Models above 70B accessed via API only allow logit lens, not interventions.

Logit lens applies the model's unembedding matrix directly to each layer's residual stream to observe how next-token predictions form across layers. Run the model with run_with_cache, extract hook_resid_post per layer, multiply by W_U, and track the target token's rank.

No. High linear probe accuracy only shows information is linearly encoded in the representation, not that the model causally uses it. Always run a random-label shuffle control to detect overfitting, and prefer causal interventions like activation patching for stronger claims.

Forward passes over large prompt sets can take over 30 minutes, so activations are saved to disk keyed by a SHA256 hash of model ID, prompts, and hook points. Cache hits and misses are logged to events.jsonl so reruns skip redundant computation.

Attention patterns show where a head attends but reveal nothing about causality. Observing a pattern is only hypothesis-generating; it must be combined with activation or path patching across sufficient examples and seeds to establish causal claims.