pipette
ESEspañol

DRAFT: Decoding and Reconstructing 3D Room Layouts from fMRI and Camera Trajectories

Chi-Wei Lee, An-Kai Chen, Cheng-Han Chiang, Yi-Chun Chang, Po-Chih Kuo

PreprintClaims a big step

In the authors' words

While functional magnetic resonance imaging (fMRI) decoders reconstruct images and 3D objects, recovering the metric geometry of rooms from a moving viewpoint remains an open challenge. We introduce, to our knowledge for the first time, the task of reconstructing 3D layouts of held-out scenes using only the known camera trajectories and the fMRI signals of participants watching indoor walkthroughs, without any visual frames. The task is hard because of the temporal mismatch between rapid visual changes and blurred fMRI signals. Our method, DRAFT (Decoding and Reconstructing 3D room lAyouts from fMRI and Trajectories), extracts a compact layout representation from the early visual cortex and fuses this neural code with the geometric free space derived from camera motion. To evaluate it, we propose layout r, the Pearson correlation between predicted and ray-cast log depths. On ten held-out scenes (20 participants), DRAFT reaches a layout r of 0.480, outperforming the camera path alone (0.446) and seven fMRI decoders (at most 0.429). DRAFT also works as a plug-in that improves all seven decoders without retraining, raising their layout r from 0.171-0.429 to 0.487-0.493. Brain activity and camera motion can thus recover the metric layout of unseen rooms, a first step toward reconstructing the 3D spaces people see.

Main resultThe abstract does not state a limitation.

In plain words AI summary

The authors created a new task: rebuilding 3D room layouts using only brain scans and camera paths, with no actual video. Their method, called DRAFT, pulls a compact room shape from early visual cortex and blends it with free space data from camera movement. On ten new scenes with 20 people, DRAFT scored 0.480, beating camera path alone (0.446) and seven other brain decoders (max 0.429). DRAFT also boosted all seven decoders without retraining, lifting their scores from 0.171-0.429 up to 0.487-0.493. Brain activity plus camera motion can thus recover the measured shape of unseen rooms.

Written by Kimi K2.5 using only the abstract above. Jev checked that every sentence is supported by the abstract sentences it cites, and removed any that were not. Point at a sentence to highlight its source.

Appeared: Friday, October 23. arXiv. Preprint, not yet peer-reviewed.

Authors' comment: 42 pages, 7 figures, 18 tables