Entropy‑weighted multilayer attention for token‑level attribution in autoregressive language models

Abstract

We present a lightweight method for token-level attribution in autoregressive transformers that uses Shannon entropy of each layer’s attention distribution as a selectivity signal for multi-layer aggregation. Low-entropy (peaked) layers, those exhibiting highly selective attention, receive higher weight; high-entropy (diffuse) layers are suppressed. The method requires a single forward pass with no gradients or masking. Evaluated on Gemma 4 E2B (2.3B parameters, 35 layers) across 400 synthetic cases spanning coreference, factual completion, syllogistic reasoning, and variable binding, entropy-weighted aggregation achieves an overall net attribution difference (NAD) of 0.358 (±0.213), compared with 0.128 (±0.518) for last-layer attention.