pipette
ENEnglish

A Geometric Invariant of Attention Predicts Head Specialization from Weights Alone

Miguel P. Bento

PreprintAfirmaciones fuertes, leer con cuidadoCódigo disponible

En palabras de los autores

Transformers are built with continuous symmetries in the attention mechanism. We derive , a scalar invariant that measures each head's capacity to couple to rotary positional encodings. This framework allows us to analyze heads without forward passes, opening a new research direction into data-independent analysis of modern LLMs. Empirically, we find that predicts induction heads, the most-studied circuit type in mechanistic interpretability, with Spearman correlation between and across four bias-free RoPE models spanning M to B parameters. As a preliminary proof of concept, we use to apply YaRN to only the top-50% positional heads, which degrades perplexity less than uniform application. Code is available at https://github.com/MPedraBento/rho-invariant.

Resultado principalLimitación que admiten los autores

Apareció: martes, 10 de noviembre. arXiv. Preprint, todavía sin revisión por pares.

Comentario de los autores: Accepted as an extended abstract at the Symmetry and Geometry in Neural Representations Workshop (NeurReps), NeurIPS 2026