pipette
ESEspañol

ReTrace: Visual Backtracking for Efficient Multimodal Large Language Models

Guangzong Si, Dong Wang, Yifan Yu, Miao Liu, Panwang Pan, Wentao Zhu

Preprint

In the authors' words

Multimodal Large Language Models (MLLMs) encode images into dense visual token sequences, creating substantial attention and KV-cache overhead. Existing visual token compression methods reduce this cost by pruning low-saliency tokens in shallow layers, but this practice couples temporary computational deactivation with irreversible information deletion. We show that this coupling breaks an intrinsic inference pattern in MLLMs. Through layer-wise attention analysis, we identify Visual Backtracking: visual pivots remain spatially stable, yet in middle layers they actively retrieve context from tokens that appeared unimportant earlier. Thus, early low-saliency tokens can be inactive for computation while still being useful for later reasoning. Motivated by this finding, we propose ReTrace, a backtracking-aware compression framework that introduces a dormant token state. ReTrace caches non-pivot tokens outside expensive layer-wise computation, restores them during the backtracking window, and prunes only after their information has been absorbed. Experiments on Qwen2.5-VL and InternVL3 demonstrate nearly 90% reductions in FLOPs and KV cache, a 2.83 prefilling speedup, and strong performance retention across complex vision-language benchmarks.

Main resultThe abstract does not state a limitation.

Appeared: Tuesday, October 20. arXiv. Preprint, not yet peer-reviewed.

Authors' comment: Accepted to Findings of EMNLP 2026, forthcoming