CNAI4S
智算平台热点追踪算子框架领域模型科学任务智算套件数据资源智算评测协作社区运行情况

友情链接

科技部工信部北京航空航天大学浙江大学
中山大学北京大学华中科技大学之江实验室
百度飞桨华为昇思中科可控
版权所有:科学智算共性平台 京ICP备18020905号-2
京公网安备11010802046602号
联系邮箱: tianyuc@buaa.edu.cn
The Alignment Illusion in Multimodal Large Language Models
返回热点追踪
新闻Astronomy

The Alignment Illusion in Multimodal Large Language Models

a
arXiv AI4S Papers
2026年9月25日技术突破
摘要

Layer-wise visual-text similarity in Multimodal Large Language Models (MLLMs) is widely interpreted as evidence that the language model progressively integrates visual content into

Layer-wise visual-text similarity in Multimodal Large Language Models (MLLMs) is widely interpreted as evidence that the language model progressively integrates visual content into a shared representation space. This reading rests on the assumption that scalar alignment scores reflect content-level cross-modal interaction. To test this assumption, we apply controlled interventions to the visual stream. Across 13 MLLMs from five families spanning 0.5B to 72B parameters, replacing projector-output visual tokens with Gaussian noise sharply reduces task accuracy, yet four standard scalar measures (CKA, SVCCA, MIR, and the leading principal-angle cosine) fail to consistently separate the corrupted stream from the original. We call this failure the alignment illusion and trace it to the shared language-model pathway: anisotropic MLP down-projections pull visual and text tokens toward common output directions, producing weight-induced alignment. Because this component is essentially one-dimensional, we introduce the principal-angle gap (PA gap), defined as the difference between the top two principal-angle cosines, which separates weight-induced similarity from multi-directional visual structure. Under graded visual corruption, the PA gap tracks task accuracy more consistently than the scalar scores we consider; under a structured but irrelevant image, it further exposes regimes in which internal geometry and task accuracy come apart. Internal visual-text alignment in MLLMs is therefore best read as a geometric diagnostic of the visual stream inside the language model rather than a direct proxy for content-level cross-modal interaction, and is most informative when calibrated by controlled task evidence.

查看原文 →

相关推荐

Temporal Gradient Inversion for Private Trajectory Reconstruction in Embodied Reinforcement Learning
新闻Astronomy

Temporal Gradient Inversion for Private Trajectory Reconstruction in Embodied Reinforcement Learning

Sudip Bhujel, Shanghao Shi, Ruiquan Huang

Molecular Beam Epitaxy of AgTaO3
新闻Astronomy

Molecular Beam Epitaxy of AgTaO3

Tobias Schwaigert, Joshua Maile, Olivia Peek

PoEM: Predicting RL Outcomes from Existing Policies
新闻Astronomy

PoEM: Predicting RL Outcomes from Existing Policies

Kimia Hamidieh, Giannis Daras, Antonio Torralba

Underwater C3-JEPA: An Object-Centric Cross-View World Model for ROV Salvage
新闻Astronomy

Underwater C3-JEPA: An Object-Centric Cross-View World Model for ROV Salvage

Yuncong Yang, Jinlong Li, Yulong Xue