pipette
ESEspañol

A Geometric Invariant of Attention Predicts Head Specialization from Weights Alone

Miguel P. Bento

PreprintBold claims, read criticallyCode available

In the authors' words

Transformers are built with continuous symmetries in the attention mechanism. We derive , a scalar invariant that measures each head's capacity to couple to rotary positional encodings. This framework allows us to analyze heads without forward passes, opening a new research direction into data-independent analysis of modern LLMs. Empirically, we find that predicts induction heads, the most-studied circuit type in mechanistic interpretability, with Spearman correlation between and across four bias-free RoPE models spanning M to B parameters. As a preliminary proof of concept, we use to apply YaRN to only the top-50% positional heads, which degrades perplexity less than uniform application. Code is available at https://github.com/MPedraBento/rho-invariant.

Main resultLimitation the authors admit

Appeared: Tuesday, November 10. arXiv. Preprint, not yet peer-reviewed.

Authors' comment: Accepted as an extended abstract at the Symmetry and Geometry in Neural Representations Workshop (NeurReps), NeurIPS 2026