Geometric knowledge spaces, Attention pairing and Architectures
En palabras de los autores
In this paper, we propose a geometric approach to deriving attention mechanisms and exploring potential new transformer architectures. We begin by formalizing the notions of knowledge worlds and attention pairings, then demonstrate how this framework naturally leads to the architectures of standard attention, causal attention, linear attention, tensor attention, and recurrent neural networks. Finally, we introduce several new geometric spaces that may inspire novel neural network architectures.
Resultado principalEl resumen no menciona limitaciones.
Apareció: jueves, 22 de octubre. arXiv. Preprint, todavía sin revisión por pares.
Comentario de los autores: 14 pages