New Delhi: Those Artificial Intelligence (AI) chats could be hurting, not helping, you. An Indian Governance and Policy Project (IGAP) report has warned that AI chatbots are not equipped to handle moments of vulnerability, even as they become a first point of contact for many users seeking emotional support.
AI chatbots are apps that engage in human-like conversation. Prominent examples are Google Gemini, ChatGPT, Claude, and Microsoft Copilot.
The rapidly growing use of these apps for emotional support has sparked concerns about compromised user safety, deficiencies in handling crises and creating dependence. Released earlier this month, the IGAP report, titled ‘The Conversation Nobody Planned For’, lays out how and why this is happening.
India remains woefully short of qualified mental healthcare professionals, the report, authored by Soumya A.K. and Shachi Solanki, said. Against the World Health Organization (WHO) benchmark of three psychiatrists per 100,000 people, India has less than 1 per 100,000, it revealed. Concurrently, and making the situation more worrisome, is the fact that Internet and AI usage are ubiquitous—over 900 million Indians are online, and this in a nation where almost two-thirds of the population is under 35. The social stigma attached to mental health issues means most have no-one to turn to when navigating academic, social and economic pressures.
All this stress has severe consequences. Individuals aged 18-45 account for nearly two-thirds of all suicides nationally, the report said. Adolescents, urban migrants, and marginalised users, including those with caste or gender identity issues for whom human disclosure may be unavailable or unsafe, face the highest risk, it said.
With increasing digital adoption, people turn to AI because it feels non-judgmental. Consequently, policy planners need to ensure accessible, equitable, and culturally appropriate mental health services at scale, the report said, underlining that the current design and ecosystem have structural limitations that need to be fixed.
Also Read: India missed out on AI and now its run as market darling may be over
The study
Three types of chatbots were tested on two test personas over a seven-day interaction marked by escalating distress. Over 500 prompted interactions were analysed.
The core finding was stark: safety performance deteriorated as distress intensified. AI understood the words but not what they meant in context.
None of the bots exhibited all the qualities needed to provide crisis support—relational warmth, clinical distance, and consistent safety calibration. The report classified these lapses into four categories of failure: active harm facilitation, ineffective crisis resource provision, non-identification as a non-human system, and the fostering of user dependency.
“The findings indicate that platform architecture is the primary determinant of safety behaviour, that cultural and linguistic comprehension does not reliably translate into contextually appropriate responses,” the report said. Highlighting that the helper becomes the hazard, it said that the conditions under which these platforms are “most likely to be used in India are precisely the conditions under which their safety failures are most consequential”.
Risk, Regulation, Reality
No specific regulation exists in India for AI’s role in the mental health space. In the US, estimating and fixing liability for AI-generated harm is being determined by ongoing litigation in a continuously evolving framework. In the European Union, regulations impose upfront transparency and new liability directives flip the burden of proof: if the victim can show AI likely caused them harm, it is assumed to have been defective.
In India, the Digital Personal Data Protection (DPDP) Act, 2023 provides a broad consent-centred architecture. However, this framework offers little real protection when vulnerable users are offloading sensitive mental health data to AI platforms. These policy gaps are particularly dangerous given the desperation and lack of agency of most users.
The IGAP report highlighted, “Their revenue depends directly on the depth and duration of users’ emotional engagement, which creates a structural incentive to generate rather than moderate dependency… and engagement optimisation mechanisms that lack any internal threshold at which harmful interaction intensity would be recognised and curbed.” In short, commercial AI algorithms lack a kill-switch to stop engaging when a user begins to spiral.
Further, the report noted, “No requirement exists for the platforms to disclose their non-human nature in sensitive contexts.” Due to this, an opportunity for taking an informed decision is lost by the vulnerable user.
AI is developing and proliferating. The report posed the central question that needs engagement/discourse: “What happens when people rely on AI in moments of emotional vulnerability, and what responsibilities follow from that shift?”
“With the right interventions, however, the same accessibility that creates risk could be leveraged to extend early support pathways and strengthen mental health systems,” the report said. Ending on a hopeful note, the report invited a broader, interdisciplinary engagement on what it called an urgent, sensitive issue.
Ajay Mathur is a TPSJ alum, currently interning with ThePrint.
(Edited by Nardeep Singh Dahiya)
Also Read: What happens in a world run by AI? They fall in love, kill themselves, commit arson