pipette
ESEspañol

Anatomy-Activated Mixture-of-Experts for 3D Medical Vision-Language Pre-training

Szymon P{\l}otka, Gizem Mert, Pedro R. A. S. Bassi, Wenxuan Li, Zongwei Zhou, Anna Marcinkiewicz, Wiktoria Roma\'nczyk, Jaros{\l}aw B. \'Cwik{\l}a, {\L}ukasz Struski, Ewa Szczurek, Jacek Tabor, Arkadiusz Sitek

PreprintBold claims, read criticallyReal-world useCode available

In the authors' words

Medical Vision-Language Pre-training (VLP) leverages the semantic richness of radiology reports to learn 3D representations, yet current architectures remain fundamentally anatomy-agnostic. While clinical diagnostics rely on organ-specific context, standard encoders apply a uniform, dense parameterization to all volumetric patches, and existing Mixture-of-Experts (MoE) routers gate on isolated local features without explicit access to the organ composition of the scan or crop. We propose Anatomy-Activated Mixture-of-Experts (AMoE), a framework that incorporates anatomical inductive biases within the model's computational path. AMoE introduces two key components: (1) a Histogram Representation Router (HRR) that conditions expert selection on global visual context and a predicted anatomical composition histogram, encouraging anatomy-dependent expert weighting; and (2) an Organ-Query Multi-Scale Projector (OQMSP) that utilizes learnable queries to extract organ-level embeddings for fine-grained alignment without requiring manual annotations at inference. By routing at the patch level rather than the token level, AMoE reduces routing complexity by orders of magnitude while satisfying load balancing by construction. Pre-trained on over 60,000 multi-phase CT scans, AMoE achieves state-of-the-art performance across segmentation, classification, and detection benchmarks. Furthermore, we demonstrate that our anatomically grounded inductive biases support robust generalization to MRI and ultrasound, establishing a scalable foundation for medical representation learning. The code is available at https://github.com/gmum/A2MoE.

Main resultThe abstract does not state a limitation.

Appeared: Tuesday, November 10. arXiv. Preprint, not yet peer-reviewed.

Authors' comment: NeurIPS 2026