• Attention Is All You Need / Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin / 2017 / https://doi.org/10.5555/3295222.3295349
• What Does BERT Look At? An Analysis of Attention / Kevin Clark, Urvashi Khandelwal, Omer Levy, Christopher D. Manning / 2019 / https://arxiv.org/abs/1906.04341
• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned / Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, Ivan Titov / 2019 / https://arxiv.org/abs/1905.09418
• Attention Is Not Explanation / Sarthak Jain, Byron C. Wallace / 2019 / https://arxiv.org/abs/1902.10186
• Attention Is Not Not Explanation / Sarah Wiegreffe, Yuval Pinter / 2019 / https://arxiv.org/abs/1908.04626
• Quantifying Attention Flow in Transformers / Samira Abnar, Willem Zuidema / 2020 / https://arxiv.org/abs/2005.00928
• Transformer Interpretability Beyond Attention Visualization / Hadar A. Chefer, Shir Gur, Lior Wolf / 2021 / https://arxiv.org/abs/2012.09838
• A Multiscale Visualization of Attention in the Transformer Model / Jesse Vig / 2019 / https://arxiv.org/abs/1906.05714
• Are Sixteen Heads Really Better Than One? / Paul Michel, Omer Levy, Graham Neubig / 2019 / https://arxiv.org/abs/1905.10650
• BERT Rediscovers the Classical NLP Pipeline / Ian Tenney, Dipanjan Das, Ellie Pavlick / 2019 / https://arxiv.org/abs/1905.05950
• Transformer Feed-Forward Layers Are Key-Value Memories / Mor Geva, Roei Schuster, Jonathan Berant, Omer Levy / 2021 / https://arxiv.org/abs/2012.14913
• Do Attention Heads in BERT Track Syntactic Dependencies? / Phu Mon Htut, Jason Phang, Yada Pruksachatkun, Samuel R. Bowman / 2019 / https://arxiv.org/abs/1911.12246
• What Does BERT Learn About the Structure of Language? / Ganesh Jawahar, Benoît Sagot, Djamé Seddah / 2019 / https://arxiv.org/abs/1906.04284
• Linguistic Knowledge and Transferability of Contextual Representations / Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew Peters, Noah A. Smith / 2019 / https://arxiv.org/abs/1903.08855
• Visualizing and Measuring the Geometry of BERT / Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda Viégas, Andy Coenen, Adam Pearce, Been Kim / 2019 / https://arxiv.org/abs/1906.02715
• Locating and Editing Factual Associations in GPT / Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov / 2022 / https://arxiv.org/abs/2202.05262
• A Mathematical Framework for Transformer Circuits / Nelson Elhage, Catherine Olsson, Nicholas Schiefer, Tendai Manzini, Nelson Minar, Tom Henighan, Chris Olah / 2021 / https://transformer-circuits.pub/2021/framework/index.html
• Toy Models of Superposition / Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Catherine Olsson, Michael Petrov, Ludwig Schubert / 2021 / https://transformer-circuits.pub/2021/superposition/index.html
• A Primer in BERTology: What We Know About How BERT Works / Anna Rogers, Olga Kovaleva, Anna Rumshisky / 2020 / https://arxiv.org/abs/2002.12327
• On Identifiability in Transformers / Gino Brunner, Yang Liu, Damián Pascual, Oliver Richter, Roger Wattenhofer / 2020 / https://arxiv.org/abs/1908.04211