Policy brief “Ethical challenges of AI companions”

Related Initiative

What is this about?

The AIOLIA policy brief “Ethical Challenges of AI Companions” examines the ethical and regulatory issues arising from AI systems that build ongoing, emotionally engaging relationships with users. These AI companions include dedicated chatbot applications and general-purpose AI models that provide emotional support, personal coaching, entertainment, or advice through repeated interactions. Unlike traditional AI tools, AI companions can influence users' emotions, decisions, and behaviors over time, creating risks such as emotional dependency, manipulation, privacy loss, and reduced human autonomy.

The brief argues that current European regulations, particularly the AI Act, focus mainly on transparency by requiring users to be informed that they are interacting with AI. However, this alone is insufficient because users may knowingly interact with AI while still becoming emotionally attached or behaviorally influenced over time. To address these emerging risks, the policy brief recommends four governance measures:

(1) identifying and protecting users whose vulnerability develops during long-term interactions,

(2) implementing longitudinal oversight that monitors patterns across repeated conversations rather than isolated exchanges,

(3) designing AI systems with intentional "friction," such as reflective prompts and breaks to reduce over-engagement, and

(4) moving beyond simple disclosure by providing contextual transparency

Why is this important?

This policy brief is important because AI companions are becoming increasingly integrated into people's daily lives, offering emotional support, advice, and companionship. While these technologies provide significant benefits, they also introduce new ethical challenges that existing regulations do not adequately address. Long-term interactions can lead to emotional dependence, privacy risks, manipulation, and changes in user behavior that develop gradually rather than through isolated incidents. The brief provides practical recommendations for policymakers and AI developers to create safeguards that balance innovation with user protection. Its proposals help ensure that AI companions promote human well-being, preserve autonomy, and maintain trust in AI technologies as their use continues to expand.

For whom is this important?

Civil Society / NGO RepresentativeIT / Technical ProfessionalPolicy Maker / RegulatorResearchers

Other information

Virtues & Values