Scaling up naturalistic narrative research: A reproducible large language model-based tool for evaluating contextual information gain
In the authors' words
Narratives are a universal part of natural human experience. As experimental stimuli, narratives allow scientists to study complex mental processes with direct relevance to everyday experience. These analyses typically require manual labeling of key event features. Such characterization of narrative stimuli are extremely labor-intensive and prone to subjectivity. Automated tools could reduce these difficulties and enable scaling to longer, varied stimuli by exposing the exact quantities computed. We address this by introducing a metric that uses open-weight LLMs to compute contextual information gain. Our metric correlates with human ratings of event importance and causal centrality, critically distinguishing the latter from semantic centrality, thus overcoming limitations of existing automated tools. The metric similarly captures shared coactivation in the default mode network, a network implicated in narrative comprehension. Taken together, these results validate an automated, empirically-grounded tool that researchers can use to identify behaviorally- and neurally-relevant computational quantities in naturalistic stimuli.
Appeared: Wednesday, November 18. bioRxiv. Preprint, not yet peer-reviewed.