How can researchers make sense of the internal reasoning pathways of transformer models, particularly the roles played by attention heads, residual streams, and feed-forward memories, so that the models' predictions can be explained, audited, and edited without sacrificing performance?
Transformer architectures entered mainstream machine learning with Vaswani et al. (2017), who showed that stacked self-attention could outperform recurrent networks on machine translation. As transformers were scaled into models such as BERT and GPT, their predictive power grew faster than the community's ability to interpret them. Early analyses by Clark et al. (2019) and Voita et al. (2019) hinted that attention heads might align with linguistic structures, inspiring probing studies by Tenney et al. (2019) and Jawahar et al. (2019). At the same time, skeptics like Jain and Wallace (2019) argued that attention weights were poor explanations, forcing interpretability researchers to broaden their toolkits. Subsequent work traced attention flow across layers (Abnar and Zuidema 2020), fused attention with gradient-based relevance (Chefer et al. 2021), interrogated BERT's geometry (Reif et al. 2019), and analyzed how feed-forward blocks store factual knowledge (Geva et al. 2021; Meng et al. 2022). Mechanistic overviews such as Rogers et al. (2020) and Elhage et al. (2021) linked these strands, while Brunner et al. (2020) highlighted identifiability limits. The present literature review sits at the culmination of that research program and synthesizes the accumulated batch notes and intermediate syntheses across these sources.
Multi-head attention is the core transformer mechanism that projects queries, keys, and values into multiple subspaces so each head can focus on different contextual relationships. Attention flow extends this idea across layers by multiplying attention matrices to estimate how influence propagates from input tokens to outputs, as in Abnar and Zuidema's work. Faithfulness refers to whether an interpretability method reflects the causal factors the model actually relies on; Jain and Wallace used counterfactual attentions to test faithfulness, while Wiegreffe and Pinter (2019) proposed sufficiency and plausibility diagnostics to defend attention-based explanations. Probing is the practice of training auxiliary classifiers on frozen representations to detect linguistic information, used heavily by Tenney et al., Liu et al. (2019), and Jawahar et al. Residual stream analysis, championed by Elhage et al., treats all transformer sublayers as additions to a shared vector, enabling circuit-level reasoning. Key-value memories describe the view that feed-forward layers store associations between activation patterns and factual values, as argued by Geva et al. and operationalized by Meng et al. Superposition, explored by Cammarata et al. (2021), names the phenomenon where multiple features share the same neurons, complicating any attempt to assign single meanings to units.
The first conversation debates whether attention visualizations explain model decisions. Clark et al. and Htut et al. (2019) show that specific heads track syntactic dependencies or coreference links, while Vig (2019) and Voita et al. provide richer visual tools and ablations that suggest specialization. Jain and Wallace, however, demonstrate that counterfactual attention distributions can leave predictions unchanged, prompting Wiegreffe and Pinter to argue for more careful criteria. The second conversation concerns representational geometry: Jawahar et al. and Reif et al. report that lower BERT layers encode lexical signals, middle layers capture syntax, and upper layers house semantics, reinforcing Rogers et al.'s broader BERTology synthesis. Liu et al. add that probe success depends on transferability, emphasizing the need to distinguish genuine knowledge from probe memorization. The third conversation explores mechanistic localization. Elhage et al.'s transformer circuits framework models how attention, MLPs, and layer norms interleave, while Cammarata et al. explain how superposition enables dense feature packing. Geva et al. and Meng et al. move from description to control by interpreting MLPs as editable memories and demonstrating interventions such as ROME. Finally, Brunner et al. stress identifiability: even if a head appears to encode a function, parameter symmetries may allow alternative realizations, so claims about circuits must heed invariances.
Attention-focused studies combine visualization, alignment metrics, and ablation. Clark et al. label tokens with syntactic roles and compute the fraction of attention mass that lands on heads aligned with those roles. Voita et al. introduce head importance scores by gating heads and noting translation degradation when key heads are removed, whereas Michel et al. (2019) prune large numbers of heads to test redundancy. Abnar and Zuidema quantify attention flow by multiplying attention matrices, and Vig builds interactive multiscale plots that aggregate head behaviors. Faithfulness critiques use counterfactual perturbations: Jain and Wallace re-optimize attention matrices to keep logits stable, while Wiegreffe and Pinter test sufficiency by constraining heads to high-weight tokens. Representation papers such as Jawahar et al., Liu et al., and Tenney et al. freeze BERT layers and train probes for part-of-speech, dependency, or semantic tasks, comparing accuracy across layers. Reif et al. project contextual embeddings into low-dimensional manifolds to show smooth trajectories for specific linguistic features. Mechanistic researchers rely on causal interventions: Geva et al. ablate neurons within feed-forward blocks and inspect nearest neighbors of activation keys; Meng et al. edit MLP weights and measure how GPT's factual responses change; Elhage et al. validate algebraic models against toy experiments; Cammarata et al. build sparse autoencoders to demonstrate superposition. Brunner et al. construct mathematically equivalent transformers by permuting heads and rescaling matrices, empirically verifying identical outputs to underscore identifiability challenges.
The heaviest disagreement is over attention-as-explanation. Jain and Wallace maintain that because one can construct adversarial attentions without altering outputs, attention weights cannot be causal explanations. Wiegreffe and Pinter respond that such counterfactuals ignore model constraints and that attention passing sufficiency tests can still justify decisions. Michel et al.'s head-pruning results intensify the debate by showing that many heads contribute little, casting doubt on interpretations that treat every head as meaningful. Another disagreement pits representational probes against identifiability critiques. Tenney et al. and Jawahar et al. interpret layer-wise probe accuracy as evidence of structured linguistic stages, but Brunner et al. contend that due to symmetries, those stages might be artifacts of the chosen parameterization. Mechanistic localization also sees tension between distributed and focused views: Cammarata et al.'s superposition implies that neuron meanings are unstable, whereas Geva et al. and Meng et al. empirically edit specific neurons or weights to change factual knowledge, suggesting at least partial localization. Residual stream advocates like Elhage et al. argue that a shared coordinate system resolves these tensions, yet definitive empirical validation remains limited.
Attention-based narratives still lack standardized faithfulness metrics acknowledged by both proponents and skeptics. Future work should combine adversarial attention perturbations, sufficiency tests, and activation patching within common benchmarks. Probing studies face the longstanding criticism that probe capacity, rather than model representation, might drive high scores; integrating probes with causal interventions such as neuron removal or factual edits could clarify what knowledge is actually used. Mechanistic frameworks promise circuit-level explanations, but identifiability proofs warn that learned functions may migrate across heads and neurons without affecting outputs, leaving open whether explanations are invariant under benign reparameterizations. Superposition research indicates that scaling exacerbates feature crowding, potentially limiting the interpretability of frontier models unless new disentanglement methods emerge. Finally, edit-based approaches like ROME succeed in narrow factual domains, yet their generality, reversibility, and safety in multi-domain deployments remain under-explored.
Across two batches of detailed notes and the intermediate synthesis, the literature converges on a nuanced understanding of transformer interpretability. Attention remains a valuable descriptive lens but only gains explanatory force when paired with counterfactual tests and multi-layer flow analyses. Representation probes reveal consistent linguistic hierarchies, yet their claims must be tempered by identifiability and probe-capacity caveats. Mechanistic approaches that treat residual streams and MLPs as structured memories offer a path toward causal editing, even as superposition and redundancy caution against simplistic stories. The field's trajectory suggests that robust interpretability will require hybrid toolchains: descriptive attention analytics to spot candidate circuits, probing and geometry to contextualize representations, and targeted interventions to verify causal roles. Building such cumulative explanations is essential for translating transformer advances into accountable, auditable, and trustworthy systems for the general reader and practitioner alike.