pipette
ESEspañol

Situating Vulnerability: An Ethical Governance Framework for Anthropomorphic AI Interaction

Hanhui Xu, Jiacheng Ji, Yuhan Wang, Haoan Jin, Jaiming Luo, Mengyue Wu

PreprintReal-world use

In the authors' words

Large language models increasingly address users in the register of care: they remember personal details, adopt a first-person emotional voice, and sustain relationships across many sessions. This anthropomorphic mode of interaction generates risks that output-level metrics capture poorly, because the same empathic utterance may console one user, entrap a second, and consolidate a delusion in a third. Governance has responded in two ways. Uniform restriction refuses sensitive requests or suppresses empathic phrasing across all outputs; group-targeted protection profiles vulnerable populations and applies stricter or prohibitive rules to them. This article argues that both fail. Uniform restriction mislocates harm in the lexical form of responses rather than in the trajectory of a relationship; group-based protection depends on a list that can never be complete, treats a fluid property as a static attribute of persons, and converts protective attention into stigmatizing labels. Drawing on philosophical work that treats vulnerability as universal, relational, and differentiated, we propose that the unit of ethical governance should be the vulnerability situation rather than the vulnerable group, develop a typology of five situations - physical, emotional, cognitive, autonomy, and relational - and specify seven principles into requirements that guide response calibration without diagnosing or persistently classifying users. The framework relocates AI safety from output control to relational safety.

Main resultThe abstract does not state a limitation.

In plain words AI summary

Large language models now talk to users like caring companions, remembering details and using emotional first-person language over many chats. This human-like style creates risks that simple output checks miss, since the same caring words may help one person, trap another, or reinforce a false belief in a third. Current governance either blocks sensitive requests for everyone or labels certain groups as vulnerable and applies stricter rules to them. The authors argue both approaches fail: blanket rules mistake word choice for relationship harm, while group lists are incomplete, treat changing states as fixed traits, and stigmatize people. They propose governing by 'vulnerability situation' instead, with five types and seven principles that guide responses without diagnosing or permanently labeling users.

Written by Kimi K2.5 using only the abstract above. Jev checked that every sentence is supported by the abstract sentences it cites, and removed any that were not. Point at a sentence to highlight its source.

Appeared: Thursday, October 22. arXiv. Preprint, not yet peer-reviewed.