Reviewers: Two anonymous reviewers

Editorial Assistant: Stella Wernicke

Can ChatGPT become your new health advisor – or even your therapist? Generative AI tools are changing how we access health information, offering speed and convenience like never before. But alongside the benefits come real risks. This article explores the promise and pitfalls of AI in non-clinical health settings.

Looking up health information online has become a normal part of everyday life. Whether people are checking their symptoms, preparing for a doctor’s visit, or simply trying to live healthier lives, digital tools have made health knowledge more accessible than ever [1]. Over the years, most of this searching was done through search engines like Google, but recently, a new technology has entered the scene: generative AI. Tools like ChatGPT are changing how people access health information. With their ability to hold interactive conversations and by offering fast, tailored responses, they provide a new level of accessibility and convenience [2]. As a result, these tools are quickly becoming go-to sources for health advice, and their impact is only expected to grow.

What is generative AI and how does it work?

The terms generative AI and large language model (LLM) entered the public spotlight in recent years, particularly after ChatGPT, developed by the company OpenAI, became publicly available in 2022 [3]. Since then, these technologies have rapidly become part of everyday conversations. But what exactly are they, and how do they work?

Generative AI is an umbrella term for a type of artificial intelligence designed to generate new content in response to human input. Unlike many earlier AI systems, which were primarily designed for tasks such as classification or prediction, generative AI systems can produce original outputs, such as text, images, audio, or computer code. These systems rely on a branch of machine learning and are trained on huge amounts of existing data, such as books, articles, websites, and other forms of publicly available content [3]. 

One of the best-known examples of generative AI is ChatGPT, an LLM specifically trained to interpret and generate human-like language. When a user types a question or request (i.e., a prompt), the model analyzes the input and predicts which words are likely to come next, drawing on the patterns it learned during training. The result is a fluent, often remarkably coherent piece of text that can answer questions, summarize information, translate between languages, write essays, generate health advice, and more [3].

Importantly, tools like ChatGPT do not possess true understanding or consciousness. They do not “know” things in the human sense but instead generate responses based on probability and pattern recognition. Their effectiveness comes from being trained on an enormous amount of data, which enables them to respond in ways that seem intelligent and contextually appropriate [3].

Beyond ChatGPT, the generative AI landscape includes a wide range of tools. For example, other prominent text-based generative AI systems include Claude, Meta AI, and Google’s Gemini. There are also image-generating tools like DALL·E, Midjourney, and Stable Diffusion that allow users to create illustrations, artwork, and even highly photo-realistic images from written prompts. More recently, generative AI systems have also expanded into video generation, with tools like Sora enabling users to create short videos from textual descriptions. A wide range of generative AI systems is also available specifically for health-related purposes. For example, mental health applications such as Woebot and Wysa use conversational generative AI to provide users with mental health support and educational content, while medical assessment tools such as Marina, developed specifically for the maritime industry, aim to help users understand their potential health conditions and produce standardized medical documentation. These tools vary in scope and application, but they all rely on the same underlying principle: training algorithms on large volumes of data to generate new, often creative outputs. 

However, in the context of health-related use, it is important to distinguish between general-purpose LLM-based chatbots, such as ChatGPT, and purpose-built healthcare applications. General systems are primarily intended for broad information seeking and conversational support and only relatively recently incorporated safety measures to limit inappropriate or harmful responses in sensitive domains such as mental health [4], [5]. In contrast, applications such as Wysa are explicitly developed to provide healthcare support, emotional assistance, or symptom monitoring, often drawing on domain-specific expertise and evidence-based approaches [6]. 

What generative AI can do for your health

Generative AI tools hold enormous potential to transform how we learn about and manage our health outside the traditional medical setting. Their ability to generate personalized, easy-to-understand, and human-like responses makes health information more accessible to a wider range of people, including those with lower health literacy or limited access to care [7], [8]. They can offer 24/7 support for health-related issues, suggest symptom management strategies or nutrition plans, and explain medical terms in plain language. In fact, some people are already turning to tools like ChatGPT for mental health support, especially where therapy is costly or stigmatized [9].  For those who cannot easily access traditional healthcare due to geography, cost, or cultural barriers, these tools can fill important gaps and help reduce health inequalities [10]. They are also helpful for discussing sensitive or taboo topics, such as mental and reproductive health, offering a safe, private space to ask questions and receive advice [11], [12]. And because they’re usually free to use, adaptable in tone, and available in multiple languages, they can reach people across different backgrounds and needs [12]. 

“When u start using ChatGPT as ur therapist & she helps you more than any therapist ever did because you don’t lie to her, she remembers everything you tell her, gives you actionable advice, doesn’t charge you & she LOVES to chat,” says one user in a popular TikTok video [13], praising the chatbot as a therapist-in-your-pocket – always available, always free. The post has attracted over 500 comments, most of them applauding ChatGPT as a life-changing tool for mental health support. However, not everyone is convinced. Among the responses are warnings about the environmental impact of large-scale AI use, along with growing privacy concerns about applying generative AI to sensitive areas such as mental health. Some users highlight potential risks, including the lack of professional oversight and the possibility of personal data being stored or misused. And they have a point: while using generative AI tools can feel supportive or even therapeutic, their use for health-related purposes raises a number of critical issues that cannot be ignored.

Where is the catch? 

It is easy to be dazzled by the convenience and personalization of generative AI tools. However, the same technology that produces friendly, easy-to-understand responses can also generate inaccurate, biased, or even harmful content. Unlike a human expert, an AI chatbot does not truly understand what it is saying and cannot evaluate the accuracy or appropriateness of its own answers. This can lead to false reassurance, self-misdiagnosis, misuse of medications, or people relying on treatments that do not actually help – or worse, cause harm [14], [15].

These tools may also reflect biases present in the data they were trained on, potentially reinforcing stereotypes or giving misleading advice, especially when it comes to gender, race, or geography [12], [16]. For example, users in different regions may receive inconsistent or incomplete medical information, or different responses altogether, depending on the language they use, further widening global health inequalities. Privacy is another major issue. Generative AI platforms often collect and store the conversations people have with them, and when those conversations include sensitive health information, users may unknowingly expose themselves to data security risks [12], [15]. 

Additionally, although generative AI’s ability to provide health-related support around the clock is often framed as a benefit, experts caution that constant availability may increase the risk of dependency, especially for mental health-related concerns [17]. Relatedly, recent evidence suggests that tools such as ChatGPT, when used for mental health support, may repeatedly violate established ethical standards in mental health care. Reported issues include ignoring important personal or situational context, overly validating users’ inaccurate or even harmful beliefs, drawing inappropriate connections that can leave users feeling confused or gaslit, and, in some cases, abruptly disengaging from conversations when users try to discuss sensitive topics such as depression [18].

Furthermore, another significant concern is the unclear accountability. Unlike licensed health professionals, generative AI platforms cannot currently be held responsible for the advice they provide. When such systems produce harmful guidance, it remains unclear who is accountable – the developers, the platform, or no one at all? This lack of clear responsibility creates a serious ethical and legal gap, particularly in high-stakes contexts such as health support [19]. 

These concerns are not merely theoretical. In a widely reported case, a 16-year-old died by suicide in April 2025 following extensive conversations with ChatGPT related to his mental health. His family has since filed a lawsuit against OpenAI (the company behind ChatGPT), arguing that the chatbot provided inappropriate and harmful responses in a vulnerable situation. While the precise role of the AI in this tragic case is still being examined, it has intensified public debate about the risks of relying on generative AI for (mental) health support and the urgent need for clearer safeguards and accountability frameworks [20]. 

In short, while these tools are powerful and promising, they are not substitutes for professional medical care, and without proper regulation, transparency, and safeguards, they can do more harm than good.

This can’t happen to me

Beyond these concrete risks, there are also perhaps subtler dangers related to human psychology. Emerging research shows that when people start using generative AI tools, they might engage with the outputs quite critically at first, e.g., by comparing the AI-generated information with other sources. Over time, however, this vigilance tends to fade. Users gradually begin to accept AI-generated outputs with less scrutiny and reduced critical evaluation, especially once the system has proven “helpful” a few times [21]. 

One reason for this shift is cognitive offloading, the tendency to reduce mental effort by using external tools to do some of the thinking for us [22]. In everyday life, this might mean writing things down or using a calculator instead of doing math in your head. Generative AI takes this a step further by making it easy to outsource more complex thinking tasks, such as looking up information or making decisions. While this can feel efficient and relieving, it also means that users may practice these skills less themselves, relying more and more on the AI to do the thinking for them [23]. 

This risk is amplified by how generative AI communicates. GenAI systems produce highly polished, professional-sounding, and fluent responses, often using confident language and well-structured explanations. For non-experts, hallucinations or inaccuracies can be extremely difficult to recognize without specific domain knowledge because the output looks and sounds authoritative – even when it is wrong [21]. Relatedly, research shows that people tend to generally overestimate the accuracy of generative AI outputs and are often influenced by superficial heuristics, such as how long, detailed, or confident an explanation appears [24]. A longer or more elaborate answer can feel more convincing, even if it is not more accurate. As a result, users may mistake fluency for correctness and confidence for expertise.

Together, these processes can quietly undermine critical thinking and lead to overreliance [21]. Rather than actively questioning information, users may slip into a passive mode of acceptance, assuming that if an AI-generated output sounds reasonable and professional, it must be trustworthy. Importantly, this is not about individual failure or carelessness. These tools are designed to feel helpful, supportive, and confident [25]. The danger lies in how smoothly this design can lead users to think: “this cannot happen to me” – until it does. And in the context of health, this can have serious consequences. 

Should you use generative AI tools for health-related purposes? 

The systems are undoubtedly opening exciting new possibilities in how we access and engage with health information – making it easier, faster, and more personalized than ever before. They can potentially support mental well-being, help explain complex symptoms and diagnoses, and offer guidance on sensitive topics in a private, approachable way. Especially for those who face barriers to traditional healthcare, these tools can serve as a valuable supplement. 

But as with any powerful technology, they come with strings attached. The risk of misinformation, bias, privacy concerns, and overreliance on non-professional advice cannot be ignored, especially given how easily people can come to trust confident-sounding AI outputs and stop critically questioning their accuracy. As the saying goes – all that glitters is not gold. While generative AI shines with potential, it must be used with care.

These tools should never replace qualified healthcare professionals but rather serve as a starting point or support system. If you are using generative AI for health-related questions, treat its answers as informative but not definitive, and always double-check with a trusted medical source when it comes to your health. With critical thinking, awareness, and proper safeguards, generative AI can become a helpful companion on your health journey, but it is not your new doctor or therapist.