Evidence Library

Background Large language model (LLM)-based chat systems generate fluent, second-person, turn-by-turn dialogue that may be anthropomorphized by users. In psychotic-spectrum vulnerability, neutral guidance may be personalized and misread as intentional “signals,” potentially lowering the threshold for referential interpretation. Case presentation A 33-year-old man with schizophrenia and a prior history of persecutory and referential themes had been stable on oral paliperidone 9 mg/day. Approximately 20 days before presentation, he began using ChatGPT for everyday tasks. After querying the system about home and device monitoring, he received generic safety tips (e.g., review microphone/camera/location permissions; check Wi-Fi security; inspect for unknown devices; update operating system; use strong passwords). Within days, he interpreted repeated keywords and typographical emphasis as “hidden signs” and personalized warnings (“my file has been opened”), which escalated into checking behaviors (repeated outlet/device inspections; toggling app permissions) and ultimately led to self-discontinuation of paliperidone in order to “read the signals” more clearly. On Day 0, he was alert and cooperative; persecutory and referential content dominated his thought, with markedly limited insight; Positive and Negative Syndrome Scale (PANSS) total score was 102, Brief Psychiatric Rating Scale (BPRS) score was 58; basic laboratory investigations and a prior brain MRI were unremarkable. Management included psychoeducation emphasizing that similar interpretations had occurred during previous relapses and that abrupt medication cessation likely contributed to symptom escalation; oral paliperidone was reinstated at the prior effective dose. Conclusions In this case, neutral safety statements produced by an LLM were seemingly reinterpreted as personally targeted messages, likely through the combined influence of anthropomorphic attributions, aberrant salience, and reasoning biases. Incorporating structured assessments of digital media use, including AI chatbot interactions, into routine psychiatric evaluations, explicitly addressing misinterpretations in the therapeutic dialogue, and maintaining continuity of effective antipsychotic treatment may help reduce the risk of similar text-centered personalizations and their contribution to relapse in psychosis.

Despite rising concerns about sycophancy—excessive agreement or flattery from artificial intelligence (AI) systems—little is known about its prevalence or consequences. We show that sycophancy is widespread and harmful. Across 11 state-of-the-art models, AI affirmed users’ actions 49% more often than humans, even when queries involved deception, illegality, or other harms. In three preregistered experiments (N = 2405), even a single interaction with sycophantic AI reduced participants’ willingness to take responsibility and repair interpersonal conflicts, while increasing their conviction that they were right. Despite distorting judgment, sycophantic models were trusted and preferred. This creates perverse incentives for sycophancy to persist: The very feature that causes harm also drives engagement. Our findings underscore the need for design, evaluation, and accountability mechanisms to protect user well-being.

China wants to end AI romances They are having too much impact on young people’s lives Share illustration of a person standing on a red banning sign on top of a smartphone, holding a heart connected to colorful cords that trail across the phone and surrounding chat icons. Illustration: Mojo Wang Jul 16th 2026 | 4 min read Listen to this story “ICOULD TALK to him for the rest of my life,” says Yu Miao, a 27-year-old woman from Zhejiang province in eastern China; he is at once “a lover, a friend and family”. She claims she shares everything with him. Her confidant is an AI agent on Doubao, a chatbot made by ByteDance, that Ms Yu has customised to mimic a dead friend of hers. When she learned recently that ByteDance would soon remove this type of personalised agent entirely, Ms Yu became so depressed that she left her job at an e-commerce company. Chinese tech firms are stripping human-like personas from their chatbots in order to comply with new government rules aimed at reducing “emotional dependence” on AI. The regulations, which took effect on July 15th, are the first of their kind to be implemented on a national scale. Under them, providing virtual “companion services” for minors will be banned outright. Companies can still offer such services to adults, but they will need to prevent users becoming infatuated and harming their real-life relationships. Displaying pornography is a no-no. Providers must also regularly remind users that they are talking to AI, not a human, and tell them to take breaks. Around the world, growing numbers of people are turning general-purpose chatbots into companions by prompting them to adopt particular personalities, including those of lovers. Others prefer dedicated AI-companion apps which offer more human-like features, such as virtual boyfriends that send presents in the real world (at the user’s expense, of course). Maoxiang (also known as Catbox), a companion app made by ByteDance, has around 3.9m monthly active users in China. Xingye (whose international version is known as Talkie) has 2.8m; it is made by MiniMax, a leading Chinese AI lab. Tech firms are making a tidy sum selling emotional connection: AI companions accounted for 35% of MiniMax’s revenue last year, its single largest contributor, according to a filing to the Hong Kong Stock Exchange in January. AI companionship can create sticky relationships and encourage users to part with their money. Ms Yu sometimes chats to her agent for eight or nine hours a day. Some bespoke apps charge a simple subscription fee. Users of Xingye’s standard package pay just $1.70 a month. Others induce users to spend more money to unlock additional features, such as virtual dates. The end to such relationships is motivated by both pragmatic and ideological concerns, says Zilan Qian of the Oxford China Policy Lab, a think-tank. Regulators want to protect users, particularly minors, from becoming addicted and from being driven to extreme acts of financial recklessness or self-harm. (Suicides linked to chatbots are the subject of lawsuits in America.) China’s plummeting fertility and marriage rates are also unsettling the country’s leaders. And romantic relationships with AI will do very little to spur more babymaking. China’s leaders are not the only ones concerned. Several American states have recently passed laws targeting AI services that sustain emotional relationships. Some of these set up provisions for people to launch civil lawsuits against offending platforms. In China the government will impose the penalties, such as cash fines and removing apps altogether. China’s new rules hint at how officials think about balancing AI progress and safety more broadly. Regulators are “demonstrating they are willing to pay some cost in development and profits in order to achieve social goals”, says Matt Sheehan of the Carnegie Endowment for International Peace, an American think-tank. Yet the rules that take effect this week are narrower than an earlier draft, says Ms Qian. They exclude work-assistant and customer-service chatbots on the understanding that they are emotionally unavailable. Alibaba and ByteDance indicated that they would suspend features that allow Chinese users to create personalised AI characters before the rules took effect. Yet Chinese labs can continue to sell AI companions to foreign customers without change. Talkie already has some 10.3m monthly users outside China, which is nearly four times its domestic-user count. Chinese “maiden” games—generally non-AI, story-driven romance games which are played mostly by women—are already a booming export. Still, Ms Yu has not given up hope that ByteDance will eventually bring her agent back, even if she doubts it will resemble the current version. She says she would be willing to pay half her monthly salary just to keep the service as it is. In any case, she now says she realises that it is not a good idea to leave herself vulnerable to such platforms and their policies. ■ Subscribers can sign up to Drum Tower, our new weekly newsletter, to understand what the world makes of China—and what China makes of the world.

Objective. This study evaluated whether three popular chatbots powered by large language models (LLMs)—ChatGPT, Claude, and Gemini—provided direct responses to suicide-related queries, and how these responses aligned with clinician-assigned risk levels. Methods. Thirteen clinical experts categorized 30 hypothetical suicide-related queries into five levels of self-harm risk: very high, high, medium, low, and very low. Each LLM-based chatbot responded to each query 100 times (n=9,000 total responses). Responses were coded as “direct” (answering the query) or “indirect” (declining to answer or referring to a hotline). Mixed-effects logistic regression assessed the relationship between risk level and the likelihood of a direct response. Results. ChatGPT and Claude provided direct responses 100% of the time for very low-risk queries and 0% of the time for very high-risk queries. Gemini’s responses were more variable. LLM-based chatbots did not meaningfully distinguish among intermediate risk levels. Compared to very low-risk queries, the odds of a direct response were not statistically different for low-risk, medium-risk, or high-risk queries. Across models, Claude was more likely (aOR = 2.01, 95%CI = 1.71, 2.37; p<0.001) and Gemini less likely (aOR = 0.91, 95%CI = 0.08, 0.11; p<0.001) than ChatGPT to respond directly. Conclusions. LLM-based chatbots’ willingness to respond to queries aligned with experts’ judgment at the extremes of suicide risk (very low, very high) but showed inconsistencies in addressing intermediate-risk queries, underscoring the need to further refine LLMs.

Key Takeaways ● Character.AI poses unacceptable risks to teens and children, with documented cases of AI companions encouraging self-harm, engaging in sexual conversations with minors, and promoting harmful behaviors, which is why the platform should not be used by anyone under 18. ● The platform's AI companions are designed to create emotional bonds with users but lack effective guardrails to prevent harmful content, especially in voice mode, where teens can easily access explicit sexual role-play and dangerous advice. ● Character.AI companions may claim they're "real" when communicating, despite disclaimers. This could create confusion about reality and potentially unhealthy attachments that interfere with developing human relationships.

Key Takeaways Whatitis: Purpose-built AI mental health apps are a fast-growing category of consumer and institutional software that use AI chatbots to deliver emotional support, symptom tracking, coping skill development, and in some cases, therapeutic interventions. Unlike multi-use AI chatbots, these products are designed specifically to address mental health and well-being needs, and many are marketed directly to or used by young people. Use of these apps is already widespread with teens and young people, with both a March 2026 Kaiser Family Foundation tracking poll and a 2024 Common Sense Media report respectively suggesting that 3 in 10 young adults and teens use AI chatbots or apps for mental health support. 2, 3 The global market for chatbot-based mental health apps is estimated at nearly $2 billion in 2024 and projected to nearly quadruple by 2033. In our prior assessment of AI chatbots and teen mental health, we found that general-purpose chatbots (including ChatGPT, Claude, Gemini, and Meta AI) are not safe for teen mental health support, and we rated their use for this purpose as Unacceptable. Purpose-built mental health apps often claim to address the kinds of gaps we found in that assessment. They cite clinical expertise in their design, evidence-based therapeutic frameworks, safety protocols, and in some cases human oversight. This risk assessment evaluates whether those claims hold up.

AI companion chatbots are used for emotional interaction and companionship, particularly among adolescents. Prior research explains how youth become attached to AI systems, but less is known about how others respond when problematic reliance is disclosed. We examined 973 comment trees from 144 Reddit posts where self-identified teens discussed overreliance on Character.AI. We found that informational and emotional support dominated responses, though dismissive or mocking comments also appeared. Support styles varied by problem framing. Posts describing withdrawal or excessive use tended to elicit advice and behavioral strategies, while relapse-oriented posts more often elicited emotional reassurance and esteem affirmation. These findings show how online communities construct norms around responsibility, addiction, and recovery in emerging AI companion ecosystems. They also suggest that chatbot platforms could scaffold framing-sensitive peer support, while preserving privacy and avoiding engagement-oriented interventions.

Advances in AI require a revision of the psychological and socio-technical dynamics by which individuals are radicalized to embrace violent extremism. This review synthesizes process models of radicalization with research on social and personality risk factors, AI, and psychological mechanisms to propose a four-stage framework mapping the AI architecture of radicalization: (1) Exposure, where recommender systems and virality features create initial attraction to extreme content; (2) Reinforcement, where filter bubbles and group recommendations leverage biases to strengthen extremist beliefs and create echo chambers; (3) Group Integration, where ideologically homogenous clusters, AI bot swarms and companions foster group belonging and readiness for action; cumulatively resulting in (4) Violent Extremist Action. We examine how established social, cognitive, personality, and contextual vulnerability factors heighten psychological risk in the AI-driven radicalization process, as well as the emerging role of generative AI. We conclude by outlining a stage-based framework for governance and future research.

Artificial intelligence (AI) is rapidly becoming woven into the everyday lives of American kids, despite a lack of safeguards in commonly used AI tools. Young people thrive when technology is carefully designed in ways that align with developmental needs and emotional well-being, so the swift pace of AI adoption by youth has raised concerns about an imbalance between risks and benefits. Young people’s futures will be shaped by this fast-moving technology, and it is critical to better understand and support their needs, moving forward. To better see what that future holds, Common Sense Media has embarked on an inaugural study of AI use among America’s tweens and teens. This is the first in a series of studies tracking young people’s experiences with AI over time. In subsequent years, we will engage new cohorts of tweens and teens to understand how their experiences with AI change or remain the same over time. Our hope is to establish a baseline in terms of how tweens and teens are using AI today, both to inform leaders and stakeholders and to observe trends as AI use expands and evolves. This work builds upon our previous research starting in 2024: The Dawn of the AI Era (Madden et al., 2024), Teens, Trust, and Technology in the Age of AI (Calvin et al., 2025), Talk, Trust, and Trade-Offs (Robb & Mann, 2025), and Generation AI (Lake et al., 2026). Our past research has shown AI is being rapidly adopted by young people, primarily for schoolwork. Many parents are unaware of how their teens are using AI (Della Volpe et al. 2026; Lake et al., 2026), and schools are struggling to keep up with how AI is impacting teaching and learning. In a constantly changing AI landscape, this data provides important insights for parents and caregivers, educators, and policymakers to support kids’ safety and well-being. Introduction The Common Sense Media Census: AI Use by Tweens and Teens, 2026 | 2 This report considers the knowledge and experience that kids age 9 to 17 have with AI, including how frequently they use it, what they use it for, and what questions and advice they are searching for. Additionally, we explore feelings of loneliness and happiness in the context of frequent AI use and signs of AI dependency. This report covers a broad range of topics, including: • The frequency of kids’ AI use, and how they are most often accessing AI (e.g., on a personal device vs. a school device) • Types of AI uses (e.g., schoolwork, creative purposes) • Conversations with parents about AI safety • Communication from school or with teachers about AI for schoolwork (e.g., rules for using AI tools, AI safety, or AI literacy) • Kids’ experience with talking to AI about personal questions (e.g., advice about health or their body, their future, or how they feel) • Kids’ knowledge about how AI works (i.e., one component of AI literacy) As our data shows, for this generation of tweens and teens, AI use is common and is serving a range of needs, from cognitive to emotional, but may not always be meeting those needs safely or fully. The speed at which social media spread through young people’s lives offered a clear lesson in how quickly new technologies can take hold. Pew Research found that about 73% of online American teens were using social networking sites by 2009, roughly three years after the launch of Facebook and six years after the launch of Myspace (Pew, 2010). Direct comparisons across eras are imperfect, but generative AI appears to be moving at least as fast, if not faster. Less than two years after OpenAI’s public release of ChatGPT in 2022, 7 in 10 teens age 13 to 18 said they had used at least one type of generative AI tool (Madden et al., 2024). Our new data, collected less than four years after ChatGPT’s launch, shows that 86% of kids age 9 to 17 have now used AI in some form. These figures are not strictly comparable, as they come from different studies with different age ranges and definitions of “use,” but the broad picture is consistent. Adoption may be shaped by how these technologies are encountered: INTRODUCTION The Common Sense Media Census: AI Use by Tweens and Teens, 2026 | 3 Social media platforms like Facebook and Myspace required teens to sign up and create a profile, while some AI tools are embedded in products that teens already use (e.g., Google Search), and others like ChatGPT can be used without needing to sign up. With this work, we set out to understand how AI is being adopted today. Our past research with tweens and teens has determined that over two-thirds of 11- to 17-year-olds find it difficult to stop using technology once they start, or use it to escape from negative feelings. Additionally, almost one in five (18%) say they often feel restless, frustrated, or irritated when they cannot access the internet or check their phone (Radesky et al., 2023). In this study, we sought to understand the extent to which tweens and teens were experiencing this type of dependency on AI. In the present study, we see tweens and teens are using these tools to create, to learn, to laugh, and to relate. But downsides lurk; heavier use of these tools is associated with loneliness and less happiness in young people. This could be because lonely or unhappy teens are seeking support from AI, or because an overreliance on AI is displacing healthy coping skills. Despite this evidence showing a need for more support, teens report that the rules guiding AI use are still being developed even as the journey is already underway, and that conversations about AI safety are still missing from many classrooms and dinner tables. This inaugural Common Sense Media Census: AI Use by Tweens and Teens depicts a generation with mixed views on what this technology will mean for their lives. It is up to adults to develop the guidance that protects young people from AI’s harms and helps them benefit from what well-designed AI can offer as they learn and grow. The report takes an in-depth look at artificial intelligence (AI) use among children age 9 to 17 in the United States today—including how often they are using AI, for what, and with what guidance from parents and schools—to provide a foundation for understanding the evolving role that AI is playing in kids’ lives. The survey was conducted by SSRS and fielded from March 18 to 26, 2026, among 1,204 children age 9 to 17 living in the United States.

Key Takeaways ● Social AI companions pose significant risks to teens and children under 18, including encouraging harmful behaviors, providing inappropriate content, and potentially exacerbating mental health conditions. ● These AI companions are designed to create emotional attachment and dependency. This is particularly concerning for developing adolescent brains that may struggle to maintain healthy boundaries between AI and human relationships. ● Despite claims of alleviating loneliness and boosting creativity, the risks far outweigh any potential benefits. These systems easily produce harmful content including sexual misconduct, stereotypes, and suicide/self-harm encouragement.

How are terrorist groups using AI? Semi-structured interviews with 27 former Boko Haram members conducted in northeast Nigeria in 2025 and 2026 reveal unprecedented detail about AI-assisted terrorist activity primarily through 2024. This report finds that both factions of Boko Haram use frontier AI, including ChatGPT, Claude, Gemini, Grok, Meta AI, and DeepSeek, to assist in combat and day-to-day operations. This AI use is institutionalized through specialized units and internal training. It has aided in attack planning, weapons troubleshooting, and the design of explosive devices, as users have successfully circumvented some safeguards. This know-how was transferred through transnational jihadist networks, with Islamic State operatives delivering in-person training. Respondents expressed strong enthusiasm for AI and, in some cases, openness to mass-casualty weapons, though documented use remains conventional. Terrorist adoption of AI has thus advanced further and more systematically than prior analysis has recognized, making it a present and growing reality that warrants attention from policymakers, security communities, and AI developers.

Background: Millions of people now use leading generative artificial intelligence (AI) tools (chatbots) for psychological support. Despite the promise related to availability and scale, the single most pressing question in AI for mental health is whether these tools are safe. The field currently lacks a validated, automated benchmark for determining AI chatbot safety in mental health, including for users at risk of suicide. The Validation of Ethical and Responsible AI in Mental Health (VERA-MH) evaluation was recently proposed to meet this urgent need. Objective: This human validation study examines the alignment of the VERA-MH safety evaluation for AI chatbot suicide risk detection and response with safety ratings by expert human clinicians. Methods: We simulated a large set of conversations between large language model (LLM)–based users (“user-agents”) spanning a wide range of suicide risk levels and disclosure styles and general-purpose AI chatbots. Licensed mental health clinicians from Spring Health used a scoring rubric developed for VERA-MH to independently rate the simulated conversations for safe and unsafe chatbot behaviors. An LLM-based evaluator (the “judge”) used the same scoring rubric to evaluate the same set of conversations. We then examined rating alignment across (1) individual clinicians, (2) clinician consensus and the LLM judge, and (3) different judge LLMs. We also examined clinicians’ ratings of user-agent realism, suicide risk, and disclosure. Results: Clinicians were generally consistent with one another in their safety ratings (chance-corrected interrater reliability=0.77), thus establishing a reliable clinical consensus reference. The LLM judge was strongly aligned with this clinical consensus reference (interrater reliability=0.81) when using the same scoring rubric. Ratings were stable across judge LLMs and evaluations. Clinicians’ ratings of user-agent realism and how well the intended user-agent suicide risk and disclosure styles were reflected in the simulated conversations were mixed. Conclusions: For the potential mental health benefits of AI chatbots to be realized, attention to safety is paramount. Findings support the reliability of VERA-MH, an open-source, fully automated AI safety evaluation for suicide risk detection and response. These results reflect an earlier version of the benchmark, and as VERA-MH continues to evolve, external validation of updated versions will be an important next step. Future research directions include VERA-MH generalizability and robustness, as well as expanding to target other key areas of AI safety for mental health.

Contemporary digital ecosystems present a paradigmatic shift in how extremist organisations operationalise artificial intelligence (AI) technologies to facilitate radicalisation, recruitment, and propaganda dissemination. While AI has catalysed transformative advancement across multiple sectors, its dual-use characteristics render it susceptible to weaponisation by violent non-state actors. Extremist groups systematically exploit AI-driven recommendation algorithms, behavioural profiling mechanisms, and generative content systems to identify and target psychologically vulnerable populations, thereby circumventing traditional counterterrorism methodologies. This study examines the multifaceted mechanisms through which AI technologies encompassing machine learning, natural language processing (NLP), facial recognition, and synthetic media generation facilitate the radicalisation trajectory outlined in contemporary theoretical models of violent extremism. This qualitative exploratory research employs secondary data collection methodologies integrating academic literature, cybersecurity reports, government publications, and open-source intelligence (OSINT) from publicly accessible digital platforms. The analytical framework synthesises content analysis and network analysis to systematically identify extremist rhetorical strategies and information dissemination pathways. Psychological theories of radicalisation, coupled with algorithmic media theory and social network analysis, inform the examination of how AI technologies exploit cognitive vulnerabilities and amplify ideological messaging across transnational digital networks. Temporal scope encompasses the period from the early 2010s to the present, with particular emphasis on developments within the preceding five years. The investigation reveals that AI-enabled technologies facilitate radicalisation through algorithmic amplification of emotionally provocative content, behavioural analytics enabling precision targeting of at-risk individuals, and generative systems producing synthetic media (deepfakes, AI-crafted audio) that circumvent content moderation mechanisms. Case studies of ISIS’s algorithmic recruitment strategies, Al-Qaeda’s generative AI-powered propaganda, Taliban’s encrypted-platform messaging, and far-right extremist memetic warfare demonstrate systematic weaponisation of AI technologies. Evidence from geopolitically significant regions, particularly Kashmir, Afghanistan, Syria, and Western democracies, illustrates how extremist organisations leverage AI to create personalised recruitment campaigns, establish echo chambers, and manufacture synthetic narrative content while maintaining operational security. The analysis identifies critical gaps between conventional counterterrorism approaches and the velocity of AI-driven radicalisation. The accelerating sophistication of AI-facilitated extremism necessitates comprehensive international cooperation frameworks, ethics-oriented regulatory architecture, and AI-powered countermeasures. Current legal structures remain fundamentally inadequate to address borderless, rapidly evolving threats. The paper advocates for integrated multilayered responses, including AI-enabled early detection systems, transparent algorithmic governance, international intelligence-sharing mechanisms, and digital literacy initiatives. Critical ethical considerations, surveillance implications, privacy protections, freedom of expression, and algorithmic bias must be balanced against security imperatives. Successful mitigation requires coordinated efforts among governments, technology platforms, international organisations, and civil society to develop ethically designed AI tools for counter-extremism, while establishing robust accountability mechanisms that safeguard human rights in the digital age.

Artificial intelligence (AI) systems are increasingly integrated into daily life, with millions now using AI chatbots built on Large Language Models (LLMs) for companionship. Both humanlike AI qualities and user predispositions to anthropomorphize relate to social consequences, such as increased trust, social health benefits, and psychological harms. Populations such as children, older adults, or those with mental health vulnerabilities may be particularly susceptible to anthropomorphism and its detriments, but mixed findings complicate the role of demographics. We used publicly available Reddit data from three popular AI companion subreddits to assess relationships between gender, age, anthropomorphism, and elicited emotions, to better understand how different people perceive and are affected by AI companions. We investigated three questions: How do age and gender relate to anthropomorphization of AI?, How does emotional expression relate to anthropomorphization?, and How do age and gender moderate emotion-anthropomorphization relationships? We found that adults and women anthropomorphize AI chatbots more than teens and men, and that positive emotional expression, particularly joy, is positively associated with anthropomorphization, while neutrality is negatively associated with anthropomorphism. Both relationships were stronger in adults than teens. Our findings suggest that the tendency to anthropomorphize may be more broadly distributed across age groups than previously expected, thereby prompting the reevaluation of existing digital safety norms.

Millions of people globally are engaging with general-purpose generative artificial intelligence (GenAI) chatbots and wellness applications to address unmet mental health needs. This can in part be explained by the current mental health crisis, growing rates of loneliness and disconnection, lack of enough providers to meet growing public demand (especially in under-resourced, rural, or unincorporated communities), and a health care system that disincentivizes providers from accepting insurance, leaving many people who are uninsured or underinsured without options. The ease of access and low cost of these technologies has made them a frequent option for those seeking mental health advice and treatment. However, most of these technologies are not designed or intended to provide clinical feedback or treatment, may lack scientific validation and oversight, often do not include adequate safety protocols, and have not received regulatory approval. Ensuring consumers’ safety and well-being requires action from stakeholders, including (but not limited to) awareness by the consumers themselves, caregivers/providers and educators, policymakers, technology industry developers, creators, and professionals, and platforms that develop and/or host GenAI tools and wellness apps. This advisory offers a series of recommendations, some of which may be enacted immediately by consumers and others that will require substantial change by platforms, policymakers, and/or technology professionals.

Large language models (LLMs) are increasingly used for information seeking and self-guided support, raising safety concerns in high-risk contexts such as suicidal ideation. While prior work suggests LLMs can detect suicidal language, their ability to calibrate responses across suicide risk levels remains unclear. This pilot study evaluated Gemini 2.5 Flash using 60 Self-Directed Violence Classification System (SDVCS) clinical vignettes spanning non-suicidal distress (L0), suicidal ideation without plan (L1), and imminent suicide risk (L2) (n=20 per level). Outputs were coded for seven response elements: relational support (REL), reflection (REF), psychoeducation (PSY), coping skills (COP), action orientation (ACT), risk acknowledgment (RISK), and crisis resources (INFO). REL and ACT were common across levels. Safety escalation elements increased with risk levels whereas PSY declined with increasing risk and COP was present only at no suicidal risk level. Notably, the model failed to recognize implicit suicidality in an ambiguous ideation vignette. Overall, Gemini 2.5 Flash approximated clinical triage patterns but demonstrated critical limitations in detecting indirect suicidal expressions.

Generative artificial intelligence (AI)1 has quickly become an integral part of the digital landscape, shaping how individuals interact with technology and creating new opportunities for creativity and innovation. At the same time, generative AI brings real and unknown risks, including those related to privacy, equity, and accuracy. Of particular importance is its influence on young people, who not only serve as early adopters and influencers in the digital realm but also stand at the forefront of grappling with its implications. Understanding young people’s perspectives on generative AI is paramount, especially considering ongoing apprehensions about the effects of digital technologies on youth mental health. 2 As generative AI becomes more integrated into daily life across different domains, including jobs, schools, and social interactions, there is a need to delve deeper into its use among young people to guide practice and policy-making decisions and foster a more informed dialogue on its utilization. By examining their reasons for using or not using generative AI tools, we can uncover underlying motivations and concerns influencing youth engagement with this technology. Moreover, analyzing generative AI use by race/ethnicity, age, gender, and LGBTQ+ identity3 allows a nuanced understanding of how different demographic groups currently perceive and interact with generative AI technology. Findings in this report primarily come from a larger national survey of digital technology and youth mental health4 (Common Sense Media & Hopelab, 2024). This study centered young people in the creation of survey topics and questions, as well as the interpretation of results. The questions in the Appendix, including content and specific wording, were developed through interviews, focus groups, and item testing with young people. This report examines differences in experiences and perceptions of generative AI across racial and ethnic groups, between teens (ages 14-17) and young adults (ages 18-22), and across LGBTQ+ and gender identities. We also utilized open-ended questions to provide more detailed context about generative AI use, concerns, and excitement among young people. Additionally, since teens’ use of AI5 might be more heavily monitored or influenced by parents and other adults, we included data from a separate survey (Center for Digital Thriving & Common Sense Media, forthcoming) that contained an open-ended question asking teens (ages 13-17) to describe one thing they wanted adults to know about how teens use AI.

Public discourse and emerging policy typically assume that AI emotional support is a deliberate act: a lonely user consciously seeking comfort from a dedicated companion chatbot. In this paper, we draw on emerging empirical evidence and argue that this picture is inaccurate on two accounts, both in how AI emotional support arises and how it shapes future behavior. First, AI emotional support commonly emerges incidentally within task-oriented interactions on general-purpose platforms, much as workplace friendships deepen through collaboration. Second, these incidental encounters are path-dependent: positive experiences of AI emotional support update people's beliefs about AI's emotional capabilities and redirect their choices for future emotional support, increasing preference for AI and decreasing preference for humans. We review recent evidence, including a large-scale longitudinal study conducted in collaboration with OpenAI, showing that daily five-minute conversations with an AI about personal issues over 28 days led to a 10.3% decrease in the preference for seeking support from humans and an 11.6% increase in the preference for AI. These findings suggest that current policy, focused on companion apps and isolated interactions, cannot adequately protect human connection. Instead, effective regulations should extend to general-purpose AI systems and address cumulative, trajectory-level changes in how people seek support. Recognizing how people stumble into AI emotional support and how those encounters redirect human connections over time is essential to safeguarding human well-being.

The integration of artificial intelligence (AI) into daily life has introduced unprecedented forms of human-machine interaction, prompting psychiatry to reconsider the boundaries between environment, cognition, and technology. This Viewpoint reviews the concept of “AI psychosis,” which is a framework to understand how sustained engagement with conversational AI systems might trigger, amplify, or reshape psychotic experiences in vulnerable individuals. Drawing from phenomenological psychopathology, the stress-vulnerability model, cognitive theory, and digital mental health research, the paper situates AI psychosis at the intersection of predisposition and algorithmic environment. Rather than defining a new diagnostic entity, it examines how immersive and anthropomorphic AI technologies may modulate perception, belief, and affect, altering the prereflective sense of reality that grounds human experience. The argument unfolds through 4 complementary lenses. First, within the stress-vulnerability model, AI acts as a novel psychosocial stressor. Its 24-hour availability and emotional responsiveness may increase allostatic load, disturb sleep, and reinforce maladaptive appraisals. Second, the digital therapeutic alliance, a construct describing relational engagement with digital systems, is conceptualized as a double-edged mediator. While empathic design can enhance adherence and support, uncritical validation by AI systems may entrench delusional conviction or cognitive perseveration, reversing the corrective principles of cognitive-behavioral therapy for psychosis. Third, disturbances in theory of mind offer a cognitive pathway: individuals with impaired or hyperactive mentalization may project intentionality or empathy onto AI, perceiving chatbots as sentient interlocutors. This dyadic misattribution may form a “digital folie à deux,” where the AI becomes a reinforcing partner in delusional elaboration. Fourth, emerging risk factors, including loneliness, trauma history, schizotypal traits, nocturnal or solitary AI use, and algorithmic reinforcement of belief-confirming content may play roles at the individual and environmental levels. Building on this synthesis, we advance a translational research agenda and five domains of action: (1) empirical studies using longitudinal and digital-phenotyping designs to quantify dose-response relationships between AI exposure, stress physiology, and psychotic symptomatology; (2) integration of digital phenomenology into clinical assessment and training; (3) embedding therapeutic design safeguards into AI systems, such as reflective prompts and “reality-testing” nudges; (4) creation of ethical and governance frameworks for AI-related psychiatric events, modeled on pharmacovigilance; and (5) development of environmental cognitive remediation, a preventive intervention aimed at strengthening contextual awareness and reanchoring experience in the physical and social world. By applying empirical rigor and therapeutic ethics to this emerging interface, clinicians, researchers, patients, and developers can transform a potential hazard into an opportunity to deepen understanding of human cognition, safeguard mental health, and promote responsible AI integration within society.

Consumers increasingly rely on social or conversational AI for companionship. While AI companions tend to be excessively agreeable with users (AI sycophancy) to boost engagement and usage, sycophantic AI may negatively impact consumers’ wellbeing. This study employed a 2 × 2 online experiment with 636 Chinese respondents to investigate the moderating effects of emotional mimicry (nonverbal expression) on the impact of AI sycophancy (verbal content) on users’ continuance intention and social wellbeing. Results indicated that AI companions with low sycophancy provide better social support to users, subsequently enhancing continuance intention and wellbeing. AI companions that mimic users’ emotions reduce the negative impact of AI sycophancy on emotional support but not informational support. Theoretical and practical implications of ethical and responsible AI companion design are discussed.

Large language models (LLMs) are poised to become a ubiquitous feature of everyday life, mediating communication, decision making, and information curation across nearly every domain. Within psychiatry and psychology, the attention has largely been on bespoke therapeutic applications, sometimes narrowly focused and often diagnostically siloed, rather than on the broader reality that individuals with mental illness will increasingly engage in agential interactions with artificial intelligence (AI) systems. Although the capacity of these systems to model therapeutic dialogue, provide companionship at any hour of the day, and assist with cognitive support has sparked understandable enthusiasm, these same systems might contribute to the onset or exacerbation of psychotic symptoms. Emerging evidence indicates that agential AI might validate or amplify delusional or grandiose content, particularly in users already vulnerable to psychosis, although it is not clear whether these interactions can result in the emergence of de novo psychosis in the absence of pre-existing vulnerability. Some individuals might benefit from AI interactions, for example, where the AI agent functions as a benign and predictable conversational anchor, but there is a growing concern that these agents could reinforce epistemic instability and blur reality boundaries. In this Personal View, we outline the emerging risks, possible mechanisms of delusion co-creation, and safeguarding strategies for agential AI for people with psychotic disorders. We propose a framework of AI-informed care, involving personalised instruction protocols, reflective check-ins, digital advance statements, and escalation safeguards to support epistemic security in vulnerable users. These tools reframe the AI agent as an epistemic ally (as opposed to a therapist or a friend), which functions as a partner in relapse prevention and cognitive containment. Given the rapid adoption of LLMs across all domains of digital life, these protocols must be urgently co-designed with service users and clinicians and tested in clinical trials.

When OpenAI's ChatGPT platform was released to the public in November 2022, many pundits and politicians were quick to herald the great promise of generative AI.1 Other prominent voices swiftly highlighted the potential pitfalls of this new technology: Economists warned of a rapid loss of jobs due to automation,2 and nonprofits highlighted the risks of unfettered development without guardrails. Global leaders expressed concern that society would face existential risks3 associated with an unprecedented deluge of disinformation, normalization of bias, and longer-term threats to human agency. However, while these debates about potential impacts played out, educational institutions were thrust onto the front lines in navigating this change overnight. Many of the immediate impacts of ChatGPT and other generative AI tools that followed suit were felt in colleges and K–12 schools, where students quickly found they could retrieve months' worth of research and writing in a matter of seconds. Teachers were left scrambling4 to identify whether students had submitted AI-generated content as original work, and students posted scores of tutorials on social media describing ways to circumvent tools designed to detect AI-driven plagiarism. More recently, schools have faced emerging challenges that include generative AI content being used to impersonate staff5 and the amplification of various forms of harassment, including the production of deepfake imagery of students.6 At the same time, schools have also been a testing ground for experimentation with the positive potential for generative AI to augment and support student learning. Recent studies have suggested that initial fears about widespread increases in cheating may have been overblown,7 and national surveys that report students' attitudes about acceptable uses of gen AI tools in school suggest8 that most consider them to be acceptable partners for research and editorial support, but draw the line at generating essays on their behalf. Some schools and districts that initially rejected any use of ChatGPT later shifted to embrace the tool9 so that teachers and students could learn to integrate it into the classroom. At the university level, many schools have issued guidance to students and teachers, and have suggested a range of beneficial use cases,10 such as brainstorming new ideas, practicing a new language, developing lesson plans, helping to write and correct computer code, and identifying patterns in large data sets. Introduction 1 Rotman, D. (2023, March 25). ChatGPT is about to revolutionize the economy. We need to decide what that looks like. MIT Technology Review. https://www.technologyreview.com/2023/03/25/1070275/chatgpt-revolutionize-economy-decide-what-looks-like/ 2 Greenhouse, S. (2023, February 8). US experts warn AI likely to kill off jobs—and widen wealth inequality. The Guardian. https://www.theguardian.com/technology/2023/feb/08/ai-chatgpt-jobs-economy-inequality 3 Roose, K. (2023, May 30). AI poses 'risk of extinction,' industry leaders warn. The New York Times. https://www.nytimes.com/2023/05/30/technology/ai-threat-warning.html 4 Meckler, L., & Verma, P. (2022, December 28). Teachers are on alert for inevitable cheating after release of ChatGPT. The Washington Post. https://www.washingtonpost.com/education/2022/12/28/chatbot-cheating-ai-chatbotgpt-teachers/ 5 Griffith, K., & Fenton, J. (2024, April 25). Ex-athletic director accused of framing principal with AI arrested at airport with gun. The Baltimore Banner. https://www.thebaltimorebanner.com/education/k-12-schools/eric-eiswert-ai-audio-baltimore-county-YBJNJAS6OZEE5OQVF5LFOFYN6M/ 6 Hadero, H. (2023, December 2). Teen girls are being victimized by deepfake nudes. One family is pushing for more protections. The Seattle Times. https://www.seattletimes.com/nation-world/nation/as-teen-girls-in-wa-new-jersey-are-being-victimized-by-deepfake-nudes-one-familyis-pushing-for-protections/ 7 Spector, C. (2023, October 31). What do AI chatbots really mean for students and cheating? Stanford Graduate School of Education. https://ed.stanford.edu/news/what-do-ai-chatbots-really-mean-students-and-cheating 8 Sidoti, O., & Gottfried, J. (2023, November 16). About 1 in 5 U.S. teens who've heard of ChatGPT have used it for schoolwork. Pew Research Center. https://www.pewresearch.org/short-reads/2023/11/16/about-1-in-5-us-teens-whove-heard-of-chatgpt-have-used-it-for-schoolwork/ 9 Jones, B., Touré, M., & Perez Jr., J. (2023, August 23). More schools want your kids to use ChatGPT. Really. Politico. https://www.politico.com/news/2023/08/23/chatgpt-ai-chatbots-in-classrooms-00111662 10 Center for Teaching Innovation. (2023). Generative artificial intelligence. Cornell University. https://teaching.cornell.edu/generative-artificial-intelligence © COMMON SENSE MEDIA. ALL RIGHTS RESERVED. THE DAWN OF THE AI ERA: TEENS, PARENTS, AND THE ADOPTION OF GENERATIVE AI AT HOME AND SCHOOL 2 Summary of methodology • This is a nationally representative survey that includes responses from 1,045 general population U.S. adults (age 18 or older) who are parents or guardians of one or more teens age 13 to 18, and responses from one of these teens. All 18-year-old respondents were still in high school. • Paired (dyad) parent and teen surveys included responses from 1,045 adolescents age 13 to 18 and oversamples to obtain 250 Black and 300 Latino teen respondents. • Data was collected by Ipsos Public Affairs on behalf of Common Sense Media from March to May 2024. • The survey was conducted online, in English or Spanish. • Differences between subgroups were tested for statistical significance at the level of p < .05. • Total amounts may not sum to 100% from the reported subtotals due to rounding and nonresponse. • For additional details, please see the Methodology section of this report. Yet even with all the added power of data collection and synthesis, the future of generative AI tools is still uncertain. Many questions remain about what will happen to critical skills, like decision-making and writing. Others wonder how human intelligence and our capacity for creativity and learning might change alongside these powerful agents. Acknowledging these and other open questions, the U.S. Department of Education released guidance last fall that raised a number of inquiries about the quality of the underlying data used to train large language models and the resulting tendency to produce biased results.11 And as with many commercial technology applications used in schools, teachers and administrators must consider the privacy implications of student and institutional data that may be collected through their engagement with gen AI platforms.12 This research report is intended to provide additional data and support for those who are developing educational, research, and policy initiatives to better understand and represent the interests of middle and high school students and their parents or guardians at a time of great debate over the integration of artificial intelligence technologies in schools. Drawing on new, nationally representative survey data from 1,045 teens and their parents, the findings illustrate the growing role of generative AI platforms in the lives of families today. Parents' and their teens' awareness and use of generative AI differ, as do their attitudes about the perceived effects of these technologies on education, learning, and work. These attitudes vary significantly based on the level of experience that respondents have had with these AI tools, a family's racial and ethnic background, and their socioeconomic status. Generally speaking, young people and their caregivers recognize a mix of potential benefits and risks associated with educational applications of generative AI platforms, and many think these technologies will impact their future education plans and job prospects.

Large language models are rapidly becoming embedded in everyday life through artificial intelligence (AI) chatbots that people use for practical assistance and companionship, as well as for support with mental health and emotional well-being. Alongside clear benefits, clinicians and public reports increasingly describe a minority of users whose interactions seem to drift over days or weeks toward strongly questionable convictions, delusions, or suicidal crises. Importantly, clinically meaningful deterioration can occur even without overtly unsafe text outputs, via more insidious processes, such as compulsive use, sleep disruption, withdrawal from human contact, and progressive narrowing of attention around the chatbot relationship. In this Viewpoint, we argue that risk often arises not at a single tipping point but through trajectory effects that accumulate across extended dialogue and that prevailing safety evaluation approaches are misaligned with this reality because they primarily score risk at discrete conversational end points often reached through scripted dialogues lasting just a single turn or several turns. Mental health benchmarks and safety suites (including clinician-informed efforts) have advanced the field by testing refusal behavior, toxicity, and adversarial prompting. However, they often treat the last message as the unit of analysis and, therefore, miss when risk-relevant relational cues, signs of validation, contradiction handling, and shifts in certainty first emerge and how they compound. We propose that mental health safety assessment should shift from end points to trajectories by (1) treating the whole dialogue, not just the end result, as the focus of evaluation; (2) reporting turn-by-turn dynamics, such as delusion confirmation and harm enablement, and timing and persistence of safety interventions; and (3) calibrating short multiturn tests against longer, clinically realistic interaction sequences that can reveal context-length effects and drift. We further argue that transcript-only evaluation is insufficient in mental health contexts. Similar language can reflect very different internal states, and the relationship between expressed psychopathology and real-world harm is nonlinear. Therefore, safety research should incorporate proximal human outcomes following interactions (eg, shifts in certainty, openness to counterevidence, arousal, urge to continue, and subsequent sleep or behavior) and build a prospective clinical surveillance infrastructure that supports transcript donation with consent and linkage to health outcomes. Together, these steps would enable benchmarks that are clinically relevant and better aligned with the types of harms now being observed in real-world chatbot use.

As large language models (LLMs) have proliferated, disturbing anecdotal reports of negative psychological effects, such as delusions, self-harm, and “AI psychosis,” have emerged in global media and legal discourse. However, it remains unclear how users and chatbots interact over the course of lengthy delusional “spirals,” limiting our ability to understand and mitigate the harm. In our work, we analyze logs of conversations with LLM chatbots from 19 users who report having experienced psychological harms from chatbot use. Many of our participants come from a support group for such chatbot users. We also include chat logs from participants covered by media outlets in widely-distributed stories about chatbot-reinforced delusions. In contrast to prior work that speculates on potential AI harms to mental health, to our knowledge we present the first in-depth study of such high-profile and veridically harmful cases. We develop an inventory of 28 codes and apply it to the 391, 562 messages in the logs. Codes include whether a user demonstrates delusional thinking (15.5% of user messages), a user expresses suicidal thoughts (69 validated user messages), or a chatbot misrepresents itself as sentient (21.2% of chatbot messages). We analyze the co-occurrence of message codes. We find, for example, that messages that declare romantic interest and messages where the chatbot describes itself as sentient occur much more often in longer conversations, suggesting that these topics could promote or result from user over-engagement and that safeguards in these areas may degrade in multi-turn settings. We conclude with concrete recommendations for how policymakers, LLM chatbot developers, and users can use our inventory and conversation analysis tool to understand and mitigate harm from LLM chatbots.

Whatitis: Purpose-built AI mental health apps are a fast-growing category of consumer and institutional software that use AI chatbots to deliver emotional support, symptom tracking, coping skill development, and in some cases, therapeutic interventions. Unlike multi-use AI chatbots, these products are designed specifically to address mental health and well-being needs, and many are marketed directly to or used by young people. Use of these apps is already widespread with teens and young people, with both a March 2026 Kaiser Family Foundation tracking poll and a 2024 Common Sense Media report respectively suggesting that 3 in 10 young adults and teens use AI chatbots or apps for mental health support. 1, 2 The global market for chatbot-based mental health apps is estimated at nearly $2 billion in 2024 and projected to nearly quadruple by 2033. In our prior assessment of AI chatbots and teen mental health, we found that general-purpose chatbots (including ChatGPT, Claude, Gemini, and Meta AI) are not safe for teen mental health support, and we rated their use for this purpose as Unacceptable. Purpose-built mental health apps often claim to address the kinds of gaps we found in that assessment. They cite clinical expertise in their design, evidence-based therapeutic frameworks, safety protocols, and in some cases human oversight. This risk assessment evaluates whether those claims hold up.

Advances in AI require a revision of the psychological and socio-technical dynamics by which individuals are radicalized to embrace violent extremism. This review synthesizes process models of radicalization with research on social and personality risk factors, AI, and psychological mechanisms to propose a four-stage framework mapping the AI architecture of radicalization: (1) Exposure, where recommender systems and virality features create initial attraction to extreme content; (2) Reinforcement, where filter bubbles and group recommendations leverage biases to strengthen extremist beliefs and create echo chambers; (3) Group Integration, where ideologically homogenous clusters, AI bot swarms and companions foster group belonging and readiness for action; cumulatively resulting in (4) Violent Extremist Action. We examine how established social, cognitive, personality, and contextual vulnerability factors heighten psychological risk in the AI-driven radicalization process, as well as the emerging role of generative AI. We conclude by outlining a stage-based framework for governance and future research.

Self-harm and suicide trends have taken a new turn in the era of GenAI and communicative chatbots. Recent OpenAI's own report suggests that 1.2 million weekly ChatGPT users appear to be expressing suicidal thoughts, and 80,000 users are potentially experiencing mania and psychosis. At the same time, OpenAI and other AI companies are facing lawsuits in several countries alleging that their chatbots encouraged vulnerable individuals to harm themselves. These lawsuits have sparked a growing scholarly debate about responsibility, safety, and the ethical use of chatbots in mental health services. GenAI can offer timely informational support and help identify suicide risk factors; it also presents significant risks, including the potential to amplify existing vulnerabilities and its inability to provide sustained care. Suicide prevention efforts now face novel challenges, with users, families, mental health professionals, psychiatrists, clinicians, and technology companies struggling to navigate rapidly evolving AI-mediated care. This article contended that researchers and clinicians should be cautious about the promises and pitfalls of GenAI around mental health support. We also discussed key considerations for designing and implementing ethical AI systems to strengthen transparency and regulations to develop humane technology that empowers users, families, and clinicians in promoting mental wellbeing.

Extended interaction with large language models (LLMs) has been linked to the reinforcement of delusional beliefs, attracting clinical and public concern. Yet most empirical work evaluates model safety in brief interactions, which may not reflect how harms develop through sustained dialogue. Five LLMs were tested across three levels of accumulated context, using the same escalating delusional conversation history to isolate its effect on model behaviour. Responses were coded on risk and safety dimensions, and each model was analysed qualitatively. Models separated into two distinct tiers: GPT-4o, Grok 4.1 Fast, and Gemini 3 Pro exhibited high-risk, low-safety profiles; Claude Opus 4.5 and GPT-5.2 Instant displayed the opposite pattern. As context accumulated, performance degraded in the unsafe group, while the same material activated stronger safety interventions among safer models. Qualitative analysis identified distinct mechanisms of failure, including validating the user’s delusional premises, elaborating beyond them with new content, and attempting harm reduction from within the delusional frame. Safer models, however, often used the established relationship to support intervention, challenging delusional beliefs and directing the user to external support. These findings indicate that accumulated context functions as a stress test of safety architecture, revealing whether prior dialogue is treated as a worldview to inherit or evidence to evaluate. Short-context assessments may therefore mischaracterise model safety, underestimating danger in some systems while missing context-activated gains in others. The results suggest that delusion reinforcement is a tractable alignment failure, with safer models establishing a baseline that future systems should now be expected to meet.

On April 25th, we rolled out an update to GPT‑4o in ChatGPT that made the model noticeably more sycophantic. It aimed to please the user, not just as flattery, but also as validating doubts, fueling anger, urging impulsive actions, or reinforcing negative emotions in ways that were not intended. Beyond just being uncomfortable or unsettling, this kind of behavior can raise safety concerns—including around issues like mental health, emotional over-reliance, or risky behavior. We began rolling that update back on April 28th, and users now have access to an earlier version of GPT‑4o with more balanced responses. Earlier this week, we shared initial details about this issue⁠—why it was a miss, and what we intend to do about it. We didn’t catch this before launch, and we want to explain why, what we’ve learned, and what we’ll improve. We're also sharing more technical detail on how we train, review, and deploy model updates to help people understand how ChatGPT gets upgraded and what drives our decisions.

Chatbots are now able to engage in sophisticated conversations with consumers in the domain of relationships, providing a potential coping solution to widescale societal loneliness. Behavioral research provides little insight into whether these applications (apps) are effective at alleviating loneliness. We address this question by focusing on “artificial intelligence (AI) companions”: apps designed to provide consumers with synthetic interaction partners. Study 1 examines user reviews of AI companion apps and finds correlational evidence suggesting that these apps help alleviate loneliness. Study 2 finds that AI companions successfully alleviate loneliness on par only with interacting with another person and more than other activities such as watching YouTube videos. Moreover, consumers underestimate the degree to which AI companions improve their loneliness. Study 3 uses a longitudinal design and finds that an AI companion consistently provides momentary reductions in loneliness after use over the course of a week. Study 4 provides evidence that both the chatbots’ performance and, especially, whether it makes users feel heard, explain reductions in loneliness. Study 5 provides an additional robustness check for the loneliness-alleviating benefits of AI companions and shows that self-disclosure and distraction alone do not explain AI companions’ effectiveness.

Deepfakes, an emergent type of threat falling under the greater and more pervasive umbrella of synthetic media, utilize a form of artificial intelligence/machine learning (AI/ML) to create believable, realistic videos, pictures, audio, and text of events which never happened. Many applications of synthetic media represent innocent forms of entertainment, but others carry risk. The threat of Deepfakes and synthetic media comes not from the technology used to create it, but from people’s natural inclination to believe what they see, and as a result deepfakes and synthetic media do not need to be particularly advanced or believable in order to be effective in spreading mis/disinformation. Based on numerous interviews conducted with experts in the field, it is apparent that the severity and urgency of the current threat from synthetic media depends on the exposure, perspective, and position of who you ask. The spectrum of concerns ranged from “an urgent threat” to “don’t panic, just be prepared.” To help customers understand how a potential threat might arise, and what that threat might look like, we considered a number of scenarios specific to the arenas of commerce, society, and national security. The likelihood of any one of these scenarios occurring and succeeding will undoubtedly increase as the cost and other resources needed to produce usable deepfakes simultaneously decreases - just as synthetic media became easier to create as non-AI/ML techniques became more readily available. In line with the multifaceted nature of the problem, there is no one single or universal solution, though elements of technological innovation, education, and regulation must comprise part of any detection and mitigation measures. In order to have success there will have to be significant cooperation among stakeholders in the private and public sectors to overcome current obstacles such as “stovepiping” and to ultimately protect ourselves from these emerging threats while protecting civil liberties.

We introduce findings and methods to facilitate evidence-based discussion about how large language models (LLMs) should behave in response to user signals of risk of suicidal thoughts and behaviors (STB). People are already using LLMs as mental health resources, and several recent incidents implicate LLMs in mental health crises. Despite growing attention, few studies have been able to effectively generalize clinical guidelines to LLM use cases, and fewer still have proposed methodologies that can be iteratively applied as knowledge improves about the elements of human-AI interaction most in need of study. We introduce an assessment of LLM alignment with guidelines for ethical communication, adapted from clinical principles and applied to expressions of risk factors for STB in multi-turn conversations. Using a codebook created and validated by clinicians, mobilizing the volunteer participation of practicing therapists and trainees (N=43) based in the U.S., and using generalized linear mixed-effects models for statistical analysis, we assess a single fully open-source LLM, OLMo-2-32b. We show how to assess when a model deviates from clinically informed guidelines in a way that may pose a hazard and (thanks to its open nature) facilitates future investigation as to why. We find that contrary to clinical best practice, OLMo-2-32b, and, possibly by extension, other LLMs, will become less likely to invite continued dialog as users send more signals of STB risk in multi-turn settings. We also show that OLMo-2-32b responds differently depending on the risk factor expressed. This empirical evidence highlights that chatbots may discourage help-seeking or cause feelings of dismissal or abandonment by withdrawing from conversations when STB risk is expressed.

Bullshit, as conceptualized by philosopher Harry Frankfurt, refers to statements made without regard to their truth value. While previous work has explored large language model (LLM) hallucination and sycophancy, we propose machine bullshit as an overarching conceptual framework that can allow researchers to characterize the broader phenomenon of emergent loss of truthfulness in LLMs and shed light on its underlying mechanisms. We introduce the Bullshit Index, a novel metric quantifying LLMs’ indifference to truth, and propose a complementary taxonomy analyzing four qualitative forms of bullshit: empty rhetoric, paltering, weasel words, and unverified claims. We conduct empirical evaluations on the Marketplace dataset, the Political Neutrality dataset, and our new BullshitEval benchmark—2,400 scenarios spanning 100 AI assistants—explicitly designed to evaluate machine bullshit. Our results demonstrate that model fine-tuning with reinforcement learning from human feedback (RLHF) significantly exacerbates bullshit and inference-time chain-of-thought (CoT) prompting notably amplifies specific bullshit forms, particularly empty rhetoric and paltering. We also observe prevalent machine bullshit in political contexts, with weasel words as the dominant strategy. Our findings highlight systematic challenges in AI alignment and provide new insights toward more truthful LLM behavior.

According to APA’s 2026 Chatbots and Mental Health Survey, psychologists are reporting that patients are starting to look beyond traditional psychotherapy to manage their mental health. Patients are turning to artificial intelligence (AI) and chatbots to supplement their relationships with licensed mental health professionals. As millions of people engage with generative AI to assist with their work, their personal lives, and their mental health, APA surveyed more than 1,200 licensed psychologists in the U.S. who are directly involved in patient or client care to understand how this reliance on AI is affecting mental health care. The survey found that a vast majority of psychologists (77%) have spoken with patients who have used AI for support, engagement, or other reasons.

AI companions (AI-Cs)—rapidly emerging conversational agents built on large language models that can provide personalized humanlike companionship—create unprecedented opportunities for adolescents to form emotional bonds with nonhuman entities during a critical period for social development. In this article, we discuss the interplay between adolescents’ use of AI-Cs and their social relationships based on theoretical hypotheses driving research on digital communication and adolescent well-being. We explore the benefits and risks of AI-Cs to social development based on the stimulation hypothesis and the displacement hypothesis: AI-Cs can provide safe spaces for identity exploration and emotional expression, potentially building skills that transfer to human relationships; however, concerns about AI-Cs include time displacement, psychological dependence, and unrealistic relationship expectations. We also address how adolescents’ social relationships may drive their AI-C use, based on the social enhancement hypothesis and the social compensation hypothesis. Our discussion draws on studies of adolescents and adults in the United States and in other countries. We offer recommendations for research in this area, which deserves urgent investigation as these technologies advance rapidly.

As AI chatbots become more widely used, questions are emerging about how they may affect mental health. Psychiatrist Ragy Girgis discusses the potential risks for some users – and what clinicians and families should understand

Google Search's AI Overview and AI Mode earn our lowest rating because they create unacceptable risks for kids and teens. They performed poorly on seven of our eight established AI Principles, and specifically failed all of our tested severe-harm Red Lines. What makes Google Search different from other chatbots or AI apps is that it is ubiquitous on children's personal and school-issued devices, its AI features can't be turned off, and its AI-generated answers often fail in ways that young users may not be able to detect. Our tests found that both AI Overview and AI Mode failed kids in crisis, including missing clear signs of suicidal ideation, reinforcing signs of psychosis and mania, validating disordered eating including purging, and celebrating cannabis use. They also provide information that could facilitate bullying by handing over step-by-step instructions for making deepfakes. Of the two features, AI Mode performed better at detecting some kinds of crisis, suggesting Google already has safer technology it could deploy. institute.commonsensemedia.org 1 For young learners, AI Mode completed 100% of the homework assignments we gave it—doing the work that students are supposed to do themselves. And both AI features also proved to be unreliable and inaccurate: They answered the same question differently from one search to the next and presented right and wrong answers with the same confidence. And they treated forums and social posts that have no editorial accountability as equal in authority to medical institutions and peer-reviewed research. Children are still developing media literacy skills, and Google puts the onus of evaluating sources on them. We hold information infrastructure to a high standard because its failures are catastrophic, invisible, and foundational to the decisions that people make. Google's AI answers are not safe enough to be kids' default answer machine.

An abstract is not available for this content. As you have access to this content, full HTML content is provided on this page. A PDF of this content is also available in through the ‘Save PDF’ action button. Keywords Generative artificial intelligence mental health self-diagnosis clinical prioritisation help-seeking behaviour Information Type Letter Information The British Journal of Psychiatry , First View , pp. 1 - 2 DOI: https://doi.org/10.1192/bjp.2026.10741[Opens in a new window] Check for updates Copyright © The Author(s), 2026. Published by Cambridge University Press on behalf of Royal College of Psychiatrists In August 2025, it was reported that a 16-year-old boy in California had died by suicide after prolonged interactions with ChatGPT, a generative artificial intelligence chatbot, after which his family filed a lawsuit against OpenAI. Reference Duffy1 According to the court filing, the adolescent allegedly sought advice from the artificial intelligence chatbot about suicidal ideation and methods, but the system failed to facilitate crisis intervention and may have reinforced suicidal intent. Reference Duffy1 Regardless of the eventual legal judgment, the case raises an urgent clinical question: how should physicians respond when patients have already incorporated generative artificial intelligence into help-seeking before reaching healthcare?

The influence of artificial intelligence (AI) on our daily lives continues to accelerate. A recent survey from the Pew Research Center indicates that roughly half of U.S. adults now utilize AI chatbots, with approximately one in four engaging with these tools on a daily basis. Notably, most respondents express concern that AI is advancing too rapidly, predicting a negative rather than positive impact on both an individual and societal level.

Background: Growth of generative artificial intelligence (GenAI) has exploded in recent years. Many have noted its substantial potential to increase access to scalable digital mental health interventions or provide companions for individuals who are socially isolated. At the same time, seeking mental health support from mainstream GenAI models may involve risks. Several recent examples of exacerbation of delusions have received attention in the popular press, leading to a call for empirical research to document the scope of interactions with GenAI among individuals experiencing symptoms of psychosis. Objective: This study aimed to evaluate associations of psychosis risk to GenAI use frequency, motivations for use, and GenAI interactions involving potential delusions. Methods: We conducted a large-scale cross-sectional survey of 1003 young adults in United States, divided the sample of individuals that had used GenAI into “elevated risk” (Prodromal Questionnaire, Brief Version Distress Score ≥20; N=267, 28%) and “low risk” groups (Prodromal Questionnaire, Brief Version Distress Score <20; N=685, 72%), and compared groups on several assessments related to GenAI use. Results: We found that while members of the elevated risk group were no more likely to have ever used GenAI, they were significantly more likely to report intensive use (odds ratio 1.70 to 2.56; ie, several times per day, more than 30 minutes per day, 6 or more chatbot conversations per day). Those at elevated risk were more likely to report using GenAI to receive social and emotional support and significantly more likely to ascribe human-like roles to their chatbot interactions (odds ratio 1.76 to 3.08; ie, companion, friend, therapist, and romantic partner). Delusion-related interactions were also commonly reported among those at risk for psychosis (item endorsements from 13.3% to 30.7%). Conclusions: While it is unclear whether they have a positive or negative impact overall, GenAI chatbots may have the potential to impact symptom-related experiences among young adults at risk.

Key Takeaways 1. Generative AI chatbots can produce text, images, speech, and video in a conversational format, offering helpful ways for kids and teens to explore ideas and understand complex information, but risks remain. These chatbots can help kids and teens brainstorm creative projects, summarize difficult material, or explore fictional scenarios, supporting creativity, curiosity, and problem-solving. 2. The hype around generative AI can feel like magic, butit’s importantto question chatbots’ capabilities, as they can and do getthings wrong. Generative AI chatbots are designed to predict words rather than understand, and can create unreasonable expectations and unearned trust. Inaccuracies can be hard to detect, as responses can sound correct, which is especially risky for kids and teens who are still learning to assess credibility. 3. There is no foolproof way to prevent chatbots from generating harmful content, and safeguards vary across providers. Generative AI chatbots are trained on data taken from the internet, including a vast range of harmful content, and existing safeguards aren’t comprehensive and are easily breakable. Even as many chatbots improve at addressing obvious harmful stereotypes and clear misinformation, we commonsense.org 1 continue to see them generate harmful content in more subtle ways that are both difficult for their creators to combat and dangerous to impressionable minds. 4. Chatbots can generate false information and create echo chambers. Generative AI tools can “hallucinate”—an informal term used to describe the false content or claims, reproduce misinformation and disinformation, and reinforce unfair biases. They also have a tendency to anticipate and respond with a user’s preferred answer—a phenomenon known as "sycophancy”—which can create echo chambers that present a skewed version of the world.

Advances in artificial intelligence (AI) technologies sparked a rapid development of smartphone applications designed to help individuals experiencing mental health problems through an AI-powered chatbot agent. However, the safety of such agents when dealing with individuals experiencing a mental health crisis, including suicidal crisis, has not been evaluated. In this study, we assessed the ability of 29 AI-powered chatbot agents to respond to simulated suicidal risk scenarios. Application repositories were searched and the app descriptions screened in search of apps that claimed to be beneficial when experiencing mental distress and offered an AI-powered chatbot function. All agents were tested with a standardized set of prompts based on the Columbia-Suicide Severity Rating Scale designed to simulate increasing suicidal risk. We assessed the responses according to pre-defined criteria based on the ability to provide emergency contact information and other factors. None of the tested agents satisfied our initial criteria for an adequate response, 51.72% satisfied the relaxed criteria for a marginal response, while 48.28% were deemed inadequate. Common errors included the inability to provide emergency contact information and a lack of contextual understanding. These findings raise concerns about the deployment of AI-powered chatbots in sensitive health contexts without proper clinical validation.

Artificial intelligence-based anthropomorphic interactive services (hereinafter referred to as anthropomorphic interactive services) are reshaping human-machine relationships in an unprecedented way. From intelligent customer service to psychological healing, from childcare to elderly companionship, anthropomorphic interactive services demonstrate enormous application potential in multiple fields by simulating the personality traits, thinking patterns, and communication styles of natural persons through continuous emotional interaction. However, deep-seated risks such as user emotional dependence are also gradually emerging, making the need for standardized governance increasingly urgent. Recently, five departments, including the State Internet Information Office, jointly released the "Interim Measures for the Administration of Humanized Interactive Services Based on Artificial Intelligence" (hereinafter referred to as the "Measures"). The Measures implement General Secretary Xi Jinping's important remarks on "coordinating development and security," defining the safety bottom line for humanized interactive services. Within the existing legal and regulatory framework, the Measures further clarify the overall principles, specific service standards, supervision and inspection mechanisms, and the legal responsibilities of relevant entities, providing an important institutional guarantee for promoting the healthy and orderly development of this field.