Ranking and Partitioning of Attention Heads for Sparse and Continual Reinforcement Learning Fine-tuning
En palabras de los autores
We study sparse and continual reinforcement learning fine-tuning of language models through attention-head ranking and partitioning. For sparse learning, we rank attention heads by gradient norm to identify a small subset of heads that can be trained effectively. For continual learning, we partition heads across tasks to reduce cross-task interference and preserve previously learned capabilities. Theoretically, we relate gradient norm to training effectiveness, and further characterize why sparse, separated head updates can remain stable. Empirically, across math, instruction-following, and coding tasks, tuning higher-ranked heads consistently outperforms tuning lower-ranked ones while often matching denser baselines, and head partitioning improves retention while preserving adaptation to new tasks.
Apareció: viernes, 16 de octubre. arXiv. Preprint, todavía sin revisión por pares.
Comentario de los autores: 26 pages, comments are welcome