Transformer Heads Looking for Order

Jasper van Doornmalen; Alexander Kozachinskiy; Corinna Mathwieser; Tomasz Steifer; Felipe Urrutia; Jos\'{e} Verschae; Przemys{\l}aw Andrzej Wa{\l}\c{e}ga

IA y aprendizaje automáticoTeoría del aprendizaje y optimización

Transformer Heads Looking for Order

Jasper van Doornmalen, Alexander Kozachinskiy, Corinna Mathwieser, Tomasz Steifer, Felipe Urrutia, Jos\'{e} Verschae, Przemys{\l}aw Andrzej Wa{\l}\c{e}ga

Preprint

En palabras de los autores

In this note, we show that the problem of checking, whether a sequence of bits is ordered, is not doable by 1-head 1-layer transformers but is doable by a 2-head 1-layer transformer. Unlike similar previous results, our results assume the model where transformers have an output MLP.

Resultado principalLimitación que admiten los autores

Leer el paper PDF

Apareció: miércoles, 23 de septiembre. arXiv. Preprint, todavía sin revisión por pares.

IA y aprendizaje automático Teoría del aprendizaje y optimización 23 de sept de 2026
Continuous Optimization for p-adic Models
We present the first method for native, continuous gradient descent for machine learning models with $p$ -adic parameters.
Julian Salazar, Dimitri Kanevsky, Matt Harvey y 2 más — arXiv
PreprintAfirmaciones fuertes, leer con cuidadoDice ser un gran avanceCódigo disponible
IA y aprendizaje automático Teoría del aprendizaje y optimización 23 de sept de 2026
xWhyL: Causal Interactive Learning
To fill this gap, we propose xWhyL, a formal framework connecting causality and XAI by learning causal models from explanations.
Nicholas Tagliapietra, Florian Peter Busch, Moritz Willig y 4 más — arXiv
Preprint
IA y aprendizaje automático Teoría del aprendizaje y optimización 23 de sept de 2026
Scalable Minimum-Volume Simplex Estimation with Non-asymptotic Analysis
We propose DeepMVSA, which re-expresses the minimum-volume principle in neural implicit form: a lightweight coordinate network generates the mixing weights and a triangular LU-type parameterization the dual simplex matrix, reducing the trainable-state memory to $O (K^{2})$ , independent of $N$ , and the cost per data pass to $O (N K^{2})$ .
Jun LI, Yanlong Guo, Zhaozhao Zeng — arXiv
Preprint
IA y aprendizaje automático Teoría del aprendizaje y optimización 23 de sept de 2026
Gap-Free Streaming PCA Beyond Rank-One Updates: Near-Optimal Rates and Applications to Differential Privacy
We give a new analysis of the ubiquitous Oja's algorithm [Oja82] for the most general, gap-free variant of this problem, where no eigengap assumptions are made on the underlying mean matrix, complemented by a nearly-matching lower bound.
Anming Gu, Syamantak Kumar, Kevin Tian y 1 más — arXiv
Preprint
IA y aprendizaje automático Teoría del aprendizaje y optimización 23 de sept de 2026
Neutral-Atom-based Quantum Optimization for Resource Allocation in NOMA Networks
In this paper, we investigate the use of neutral-atom quantum platforms to solve the maximum access problem (MAP), formulated as a mixed-integer programming task that jointly considers admission control, user clustering, channel assignment, and power allocation in a non-orthogonal multiple access (NOMA)-enabled uplink network.
Patatchona Keyela, Remon Polus, Soumaya Cherkaoui y 1 más — arXiv
PreprintUso en el mundo real

Todos los papers nuevos de Teoría del aprendizaje y optimización

Transformer Heads Looking for Order

En palabras de los autores

Más de Teoría del aprendizaje y optimización ese día

Continuous Optimization for p-adic Models

xWhyL: Causal Interactive Learning

Scalable Minimum-Volume Simplex Estimation with Non-asymptotic Analysis

Gap-Free Streaming PCA Beyond Rank-One Updates: Near-Optimal Rates and Applications to Differential Privacy

Neutral-Atom-based Quantum Optimization for Resource Allocation in NOMA Networks