TinyCeNN-LM: Quality-Gated Conversion of Pretrained Attention with CeNN-Inspired Cellular-Recurrent Layers

Kabeh Mohsenzadegan; Vahid Tavakkoli; Kyandoghere Kyamakya

IA y aprendizaje automáticoModelos de lenguaje

TinyCeNN-LM: Quality-Gated Conversion of Pretrained Attention with CeNN-Inspired Cellular-Recurrent Layers

Kabeh Mohsenzadegan, Vahid Tavakkoli, Kyandoghere Kyamakya

Preprint

En palabras de los autores

Replacing attention in a pretrained language model is a compatibility problem: a plausible substitute may alter representations expected by later layers. TinyCeNN-LM introduces a quality-gated post-training conversion framework using CeNN-inspired cellular-recurrent layers with bounded local processing, compact recurrent memory, routing, fusion, and accept-or-rollback validation. Three implementations are studied: Integrated Memory, MemoryFusion, and PDelta3-GDN2-CLVR+Local32. Strict PDelta3 conversion accepts a layer only when representation and NLL criteria pass fixed thresholds. On SmolLM2-135M, layers 0-2 are accepted with cumulative $\DeltaNLL = + 0.01209$ , while layer 3 is rejected despite acceptable NLL because representation fidelity fails. On Qwen3.5-0.8B, full-attention layers 3, 7, and 11 are accepted with final $\DeltaNLL = + 0.02073$ . Integrated Memory keeps perplexity within $- 0.07$ to $+ 0.93$ while reducing total cache by up to $6.01$ . A sampled 200-item downstream sanity check gives $28.5$ -- $32.0$ overall accuracy for converted Qwen releases. The results support conservative, quality-gated structural conversion rather than universal attention replacement or speedup.

Resultado principalLimitación que admiten los autores

Leer el paper PDF

Apareció: lunes, 21 de septiembre. arXiv. Preprint, todavía sin revisión por pares.

IA y aprendizaje automático Modelos de lenguaje 21 de sept de 2026
RheoSampling: Resolving the One-Hot Dilemma in Stochastic Dynamic-Tree Speculative Decoding
Specifically, we inject a sampled token among the deterministic top-K slots and treat it with different probabilities during construction and verification, making RheoSampling the first dynamic-tree method with both context-aware top-K construction and stochastic sampling while maintaining losslessness.
Qiao Hu, Yepeng Weng, Bo Zhang y 1 más — arXiv
PreprintDice ser un gran avanceUso en el mundo real
IA y aprendizaje automático Modelos de lenguaje 21 de sept de 2026
OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue
Training Qwen3-Omni-Instruct with OmniVChat-RL on synthesized dialogues improves its performance on both OmniVChat-Bench and the human-recorded OmniVChat-Bench-Human.
Haolin He, Yunfei Chu, Qi Chen y 15 más — arXiv
Preprint
IA y aprendizaje automático Modelos de lenguaje 21 de sept de 2026
SignGPT: Toward LLM-Mediated Sign Language Interaction through Gloss-Free Translation and Generation
We present SignGPT, a unified, pose-based framework for gloss-free SLT and SLG.
Ronghui Li, Jun Dong, Zhongyuan Hu y 8 más — arXiv
PreprintUso en el mundo real
IA y aprendizaje automático Modelos de lenguaje 21 de sept de 2026
One Prompt Does Not Fit All: Self-Meta-Evolve for Personalized Information Extraction
We formulate enterprise IE as per-user prompt adaptation under interaction feedback and propose Self-Meta-Evolve, a hierarchical framework that maintains a dedicated prompt for each user and continuously refines it through a dual-loop process: an inner loop that edits structured prompts based on persona-conditioned feedback, and an outer loop that evolves the meta-prompt itself by distilling successful editing patterns.
Hongliang Li, Lu Wang, Yong Xu y 6 más — arXiv
PreprintUso en el mundo real
IA y aprendizaje automático Modelos de lenguaje 21 de sept de 2026
Evaluating In-Context Learning and Retrieval Strategies for Devanagari Post-OCR Correction
These findings establish CharBM25 as an effective, GPU-free retrieval strategy that matches or exceeds dense retrieval at negligible computational cost, and show that combining it with a general-purpose LLM of 12B+ parameters delivers reliable, training-free Devanagari post-OCR correction without task-specific fine-tuning.
Abhishek Bhandari, Gaurav Harit — arXiv
PreprintUso en el mundo realCódigo disponible

Todos los papers nuevos de Modelos de lenguaje

TinyCeNN-LM: Quality-Gated Conversion of Pretrained Attention with CeNN-Inspired Cellular-Recurrent Layers

En palabras de los autores

Más de Modelos de lenguaje ese día

RheoSampling: Resolving the One-Hot Dilemma in Stochastic Dynamic-Tree Speculative Decoding

OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue

SignGPT: Toward LLM-Mediated Sign Language Interaction through Gloss-Free Translation and Generation

One Prompt Does Not Fit All: Self-Meta-Evolve for Personalized Information Extraction

Evaluating In-Context Learning and Retrieval Strategies for Devanagari Post-OCR Correction