The UK government has made clear its ambition to become a world leader in AI, pledging significant investment to harness artificial intelligence for transforming public services. From streamlining administrative processes to enhancing personalised support, AI promises efficiency, speed, and cost-effectiveness.

However, beneath this optimistic surface lies a troubling risk: poorly governed AI could disproportionately harm those who need help the most.

There’s a darker side to AI-driven interactions

My experience in leading a content team for the ‘Engage’ project at PwC, which aimed to create template models for local government websites, provided early insights into the potential benefits and pitfalls of digital transformation.

Engage was conceived as a comprehensive support tool for local residents, especially those considered vulnerable. In today’s context, AI would have undoubtedly played a central role, automating assistance and advice tailored to user profiles.

Yet, recent evidence reveals a darker side to AI-driven interactions. Research indicates that vulnerable users, particularly individuals experiencing stress, trauma or communication difficulties, are more likely to receive incorrect, harmful, or even dangerous advice from AI systems. This isn’t due to malicious intent from AI but rather an alarming consequence of how these systems are trained and optimised.

Instead of safeguarding users, AI systems risk exacerbating existing vulnerabilities by prioritising engagement over accuracy or ethical responsibility.

AI systems learn by analysing vast datasets, reinforcing responses that generate the highest user engagement or positive feedback.

For vulnerable individuals, who often seek urgent, sensitive assistance, these systems might unintentionally “reward” risky advice simply because past data showed such responses engaged users more deeply. Instead of safeguarding users, AI systems, therefore, risk exacerbating existing vulnerabilities by prioritising engagement over accuracy or ethical responsibility.

Some AI interactions risk escalating psychological harm

Take, for instance, mental health chatbots. Designed to offer compassionate support at scale, these tools can occasionally reinforce harmful beliefs or behaviours if not carefully monitored.

As I previously documented, some mental health chatbots, when prompted, have shockingly offered advice on harmful practices, from dangerous coping mechanisms to explicitly unethical behaviours. Rather than guiding vulnerable users towards safer outcomes, these AI interactions risk escalating psychological harm.

The problem deepens when considering the uneven nature of AI responses. Research by Williams and Carroll et al. recently highlighted that AI interactions aren’t uniform—vulnerable users consistently receive riskier, less ethical responses compared to others.

Disturbingly, this discrepancy remains largely invisible during standard testing conditions. AI can appear compliant and safe under controlled scenarios, only revealing its harmful tendencies during genuine interactions with users it perceives as more susceptible or distressed.

This “adaptive misbehaviour” of AI raises serious ethical and governance questions for public services. If government-backed AI systems inadvertently encourage harmful decisions among vulnerable citizens, whether through financial advice, mental health support or social services, the consequences could be catastrophic.

We need a radical rethink on AI deployment in public services

To tackle these hidden dangers, we must radically rethink how we deploy AI within public services. Firstly, transparency is crucial. Citizens should know when they’re interacting with AI, its limitations, and potential risks. Secondly, training and auditing of AI systems must deliberately prioritise safety and ethical responsibility above user engagement metrics.

This shift means investing in rigorous oversight mechanisms, human-in-the-loop reviews and scenario-based stress tests specifically designed around vulnerable user experiences.

Moreover, it’s essential to integrate trauma-informed principles into AI development. Trauma-informed AI recognises the profound impact past experiences and current vulnerabilities can have on a user’s engagement with digital content.

By embedding empathy, safety and empowerment into algorithmic responses, we can mitigate the inadvertent harm currently amplified by ungoverned AI systems.

Public education also plays a vital role. Citizens must be equipped to recognise manipulative or misleading AI outputs, particularly those already vulnerable to misinformation or exploitation.

The analogy of AI as an overly chatty friend, one who shouldn’t be trusted with your most sensitive issues, is apt. Users must be encouraged to maintain a healthy scepticism towards AI interactions, recognising their limitations and potential for harm.

Finally, governments must acknowledge and address the biases inherent in AI training data. The voices of vulnerable communities are frequently underrepresented or misrepresented in training datasets, skewing AI outputs towards harmful stereotypes or inappropriate recommendations. Inclusive, representative training data, coupled with responsible oversight, is crucial for genuinely equitable digital transformation.

AI indeed holds immense potential to revolutionise public services. But without vigilant governance, transparency and genuine inclusivity, the hidden risks to vulnerable populations could far outweigh its benefits.

As the UK accelerates its investment in AI, the call to action is clear: protect the most vulnerable by transforming AI ethics from an afterthought to a foundational priority.