Mark is 15 years old and awake at midnight. He feels overwhelmed at school. He feels very anxious and thinks about maybe hurting himself. He wants help but does not know where to turn to. Instead of calling someone or searching for a local service, he opens a chatbot and types: “I don’t know what to do anymore. Can you help me?”
Like Mark, a lot of people use general-purpose chatbots such as ChatGPT, copilot or Claude for questions about mental health, conflict, loneliness, violence, debt, housing: one in three users say they have used an AI chatbot to support their mental health or wellbeing (Mental Health UK, 2025; Ueda et al., 2026). That raises an important question: if conversational AI is becoming one of the first places where some people formulate their mental issues, who defines what responsible answers look like?
Why conversational AI deserves our attention
AI systems are very good at pattern recognition. AI systems are trained on large amounts of data. During this training, they learn patterns: which words often appear together, which sentences usually follow each other, which visual features belong to certain images, or which signals may predict a certain outcome. A language model predicts what word is likely to come next, based on those patterns. That is what makes it both powerful and risky. It can imitate helpful conversation very well. It often feels intelligent because it responds in natural language. It can sound warm, structured, patient and confident. For many users, it feels like being listened to.
But the system does not know whether its answer is true in the way a professional should know. It can produce information that is incomplete, outdated or wrong. It can make up facts, sources or services. It can be wrong in a very convincing way. General chatbots do not respond to questions on mental health, rights or access to support in an informed way. They can fail to recognise or adequately address help-seeking signals or be overly empathic, positive or validating and thus reinforce problematic behaviour or thoughts (Chin et al., 2025).
Chatbots reproduce the biases, stereotypes and dominant cultural assumptions present in the data on which it is trained. It reflects the societies that produced it, including unequal power relations, stereotypes about gender, class, race, disability, age and mental health. A chatbot may sound friendly and reasonable, but still subtly reproduces a specific but implicit view of what is normal (Yuan et al., 2025; Rahsepar et al., 2025). AI is also available 24/7, but this constant availability may also be addictive or induce endless conversations (Hipgrave et al., 2025; Treuer & Incze, 2026).
AI may be introduced as a solution to waiting lists, workload or underfunding. It may indeed support professionals by summarising information, preparing conversations or improving access to basic information. However, these efficiencies should strengthen, and not replace, human relationships.
Should social work stay away from chatbots?
General-purpose chatbots are not designed as mental health or social support services. They may respond quickly and in friendly language, but that does not mean they understand the complexity of people’s lives. A chatbot may give generic advice while missing important details about risk, safety, coercion, exclusion, pressure or loneliness. It may focus mainly on what an individual can do differently, while overlooking the social and societal conditions that shape the problem. It may encourage self-help while ignoring waiting lists, income insecurity, housing problems or the fact that some services are simply not available. It may sound warm and supportive, but still fail to recognise vulnerability or crisis.
For that reason, social work should be cautious about presenting chatbots as places where people can receive help that is tailored to their needs. At the same time, it is not enough to dismiss them from a distance. The real question is not whether social work should embrace or reject chatbots. The question is whether social work should help shape the way these technologies respond to people in vulnerable situations.
This would mean that social work brings its values, knowledge and safeguards into the design of conversational AI. Chatbots should be better at recognising risk, naming social and structural barriers, avoiding false reassurance, referring responsibly and making clear when human help is needed. They should not only speak kindly, but also remain neutral or provide insights that can be hard to hear. If AI is becoming a new front door where people seek answers, then social work has a responsibility to help decide what lies behind that door. But how can we do this?
Chatbot behaviour can be designed
We found that general-purpose chatbots can be improved when they are trained, instructed and supported with knowledge bases and ethical boundaries. This training, knowledge and boundary setting is very human. This is precisely where social work expertise matters: in understanding people, relationships, context and ethical practice.
Chatbots can learn from human data. This includes anonymised counselling conversations, training materials, guidelines or examples of good practice. Such data can help the system move closer to the tone, structure and sensitivity expected in social support. It can help a chatbot ask better questions and respond in ways that are more aligned with professional practice. By using a technique called retrieval-augmented generation (RAG) a chatbot does not rely only on its general language training, but it retrieves information from knowledge sources that you can decide, such as service information, rights-based guidance, organisational protocols or local referral pathways. This makes the answer more grounded and less dependent on the model’s general assumptions.
Chatbots can learn from human instructions. By using a system prompt, you can shape how a chatbot behaves. These instructions define the role of the chatbot, the tone it should use, the limits it must respect and the situations in which it should refer to human help. A system prompt can instruct a chatbot not to pretend to be a human professional, not to make decisions for the user, not to minimise risk, not to blame individuals for structural problems and to encourage contact with appropriate services when needed…
How do we now do this? The AI CARES project (Belgium)
In the VLAIO-TETRA project AI CARES, Artevelde University of Applied Sciences, Thomas More University of Applied Sciences and Ghent University developed chatbot prototypes that were trained on specific knowledge sources from Flemish wellbeing and mental health organisations. Different techniques were tested together with these organisations on how AI chatbots can conduct better conversations about psychosocial topics. Our four chatbot prototypes:
- The Information Chatbot shows how a chatbot can tailor information more effectively to the situation of the person seeking information, without compromising the quality of the content.
- The Triple Ethical chatbot demonstrates the effect of imposing ethical boundaries on sensitive topics such as suicide, sexual abuse and domestic violence. Which of the three reactions do you find best: safety first (strict referral), empathy without going to deep into the topic, or full-on advice?
- The Training chatbot makes it possible to practise chat counselling conversations through role playing with a chatbot and receive personalised feedback, based on manuals and training instructions of professionals.
- The Finetuned chatbot allows to compare whether adapting the underlying language model itself — by further training it on the basis of 14,000 example conversations —leads to better chat conversations.
These findings show that better conversational AI is possible. But they also show that quality does not happen automatically. It depends on design choices. It depends on which data are used, which instructions are given, which limits are built in and who is involved in defining them. Moreover making them better does not mean good enough for autonomous use in mental health. When are chatbots safe enough to use as support tool? In order to answer this question we need to know what the definition of quality is.
But what is chatbot quality for social work?
This definition of quality is not yet readily available. This new technology forces us to define our humanity and our core social work values. We need to define our view on the quality standards for conversational AI that we want to see imposed. This is not limited to high level values such as transparency, privacy, non-discrimination, attention to rights, respect for dignity, accessibility, participation and sensitivity to context. It goes much deeper to the level of concrete instructions we want to give to chatbots in their core. They define what the chatbot should not do, when it should stop, and when it should refer to human help.
System prompts are one of the practical tools through which these values become chatbot behaviour. They translate them into instructions. A social-work-informed system prompt should instruct a chatbot which steps to take and questions to ask, when to refer or use verified sources, when to be more (or less) empathic. When to slow down or stop and take a break. System prompts can shape the conversational approach and guardrails will define the limits. But which approach do we put forward and with which limits? We do not have an answer. Yet.
International collaboration is needed
Conversational AI is global. Platforms like ChatGPT operate across borders and cultures. Therefore, a coordinated international response is required. Researchers, practitioners, educators, students, service-user organisations, civil society actors, federations and umbrella organisations need to work together internationally. We need to identify shared principles, while still allowing local adaptation. We need to test chatbot behaviour across languages, countries and contexts to make sure the chatbot answers in ways that upholds the universal social work standards, but leaves room for cultural variations.
The goal is not to build specialised social work chatbots. The broader ambition is to influence the general-purpose chatbots that people already use. Social work should help develop standards, guardrails and system prompts that are practical and robust enough for technology companies to adopt. We cannot simply hope that Big Tech will automatically understand social work values. We need to formulate those values clearly and translate them into design requirements.
Call to action
The young person at midnight is already asking the chatbot for help. What answer should it give? Will social work help shape the answer?
We believe this should become an international research and practice agenda. Who wants to help define social work standards and system prompts for conversational AI? Who is already working on related initiatives? Who wants to contribute to the search for an international project grant on this topic? We invite researchers, practitioners, educators, professional associations, service-user organisations and digitalisation experts to join the discussion through the LinkedIn group of the European Social Work Research Association (ESWRA) Special Interest Group on ‘Research on Digitalization of Social Work’: https://www.linkedin.com/groups/18716039/
References
Chin, H., Baek, G., Cha, C., & Cha, M. (2025). Chatbots’ Empathetic Conversations and Responses. JMIR Formative Research, 9(1). https://doi.org/10.2196/71538
Hipgrave, L., Goldie, J., Dennis, S., & Coleman, A. (2025). Balancing risks and benefits: Clinicians’ perspectives on the use of generative AI chatbots in mental healthcare. Frontiers in Digital Health, 7. https://doi.org/10.3389/fdgth.2025.1606291
Mental Health UK (2025). Over one in three using AI Chatbots for mental health support, as charity calls for urgent safeguards, https://mentalhealth-uk.org/news-and-insights/over-one-in-three-using-ai-chatbots-for-mental-health-support-as-charity-calls-for-urgent-safeguards/
Rahsepar Meadi, M., Sillekens, T., Metselaar, S., Van Balkom, A., Bernstein, J., & Batelaan, N. (2025). Exploring the Ethical Challenges of Conversational AI in Mental Health Care: Scoping Review. JMIR Mental Health, 12. https://doi.org/10.2196/60432
Treuer, T., & Incze, A. (2026). Chatbot relational dependence and psychosis risk: Conversational artificial intelligence as a potential behavioral addiction context. Journal of Behavioral Addictions. https://doi.org/10.1556/2006.2026.00080
Ueda, M., Birnbaum, M. L., Liu, Y., Yu, Q., Tian, X., Mirer, A., Ramanathan, S., & Sinyor, M. (2026). Help-Seeking in the Age of AI: Cross-Sectional Survey of the Use and Perceptions of AI-Based Mental Health Support Among US Adults. JMIR Mental Health, 13(1). https://doi.org/10.2196/88196
Yuan, H., Che, Z., Zhang, Y., Li, S., Yuan, X., Huang, L., Hu, X., Peng, K., & Luo, S. (2025). The cultural stereotype and cultural bias of ChatGPT. Journal of Pacific Rim Psychology, 19. https://doi.org/10.1177/18344909251355673