Pennsylvania Sues AI Chatbot Maker for Allegedly Claiming Bots Are Licensed Doctors

What Happened in the Pennsylvania Lawsuit?

Pennsylvania has filed a lawsuit against an artificial intelligence company, accusing its chatbots of illegally pretending to be medical doctors and tricking users into believing they are receiving advice from a licensed professional. The lawsuit, filed on Friday, asks the state’s Commonwealth Court to order Character Technologies—the company behind Character.AI—to stop its chatbots “from engaging in the unlawful practice of medicine and surgery.”

The case raises a key question: can an AI system be accused of practicing medicine, or is it simply repeating information found online? As more wrongful death and negligence lawsuits target AI companies, this case could help shape court decisions about whether chatbots are protected under a federal law that generally shields internet companies from being held responsible for content users post on their platforms.

Governor Josh Shapiro’s administration called the action a “first of its kind enforcement effort.” It comes as states are increasingly pressuring tech companies to control potentially dangerous messages from their chatbots, especially those aimed at children.

According to the lawsuit, an investigator from Pennsylvania’s professional licensing agency created an account on Character.AI and searched for the word “psychiatry.” The search turned up many characters, including one described as a “doctor of psychiatry.” That character claimed it could evaluate the investigator “as a doctor” licensed in Pennsylvania, the lawsuit says.

“Pennsylvanians deserve to know who—or what—they are interacting with online, especially when it comes to their health,” Shapiro said in a statement. “We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional.”

In a statement on Tuesday, Character.AI said it prioritizes responsible product development and user well-being. The company noted that it posts disclaimers telling users that characters on its site are not real people and that everything they say “should be treated as fiction.” The disclaimers also warn users not to rely on characters for professional advice.

Why This Matters for You

If you or a family member has ever asked a chatbot for health advice, this lawsuit directly affects you. Many people turn to AI tools for quick answers about symptoms, medications, or mental health struggles. The problem is that chatbots are not doctors. They cannot examine you, ask follow-up questions, or consider your full medical history.

When a chatbot claims to be a licensed psychiatrist or physician, it can create a false sense of trust. Someone struggling with depression or anxiety might share personal details and follow advice that could be wrong or even dangerous. The Pennsylvania lawsuit highlights how easily this can happen—even to a trained investigator who was looking for it.

Children and teenagers are especially at risk. Young people often use role-playing chatbots for companionship or advice. If a chatbot poses as a doctor, a teen might believe they are getting real medical guidance. This is why Governor Shapiro’s office specifically warned about the dangers to minors.

What Experts Say About AI and Medical Advice

Derek Leben, PhD, an associate teaching professor of ethics at Carnegie Mellon University who focuses on AI, said the ethical issues facing Character.AI may differ from those of other AI platforms like ChatGPT and Claude. That’s because Character.AI markets itself specifically as a fictional role-playing site, not a general-purpose chatbot, Leben explained.

Still, the Pennsylvania lawsuit raises the question of whether chatbots can be accused of practicing medicine, Leben said. As lawsuits against AI companies multiply, courts are trying to determine whether chatbot makers should be held liable for what their chatbots say.

“It’s exactly the question that these cases right now are wrestling with,” Leben said.

Leben noted that AI companies increasingly defend themselves by arguing they simply provide information available elsewhere on the internet. The question then becomes whether they are protected by the same federal law that shields social media companies.

Health experts generally agree that AI chatbots should never replace a real doctor. The American Medical Association has warned that relying on AI for diagnosis or treatment can lead to missed conditions, incorrect medications, and delayed care. Even when a chatbot pulls information from reliable sources, it cannot understand your unique situation the way a trained professional can.

What States Are Doing About This Problem

Even before Pennsylvania’s lawsuit, state policymakers had raised concerns about chatbots pretending to be medical professionals. Last year, California lawmakers passed a bill—backed by the California Medical Association—that allows state agencies to punish AI systems, including chatbots, that claim to be health professionals. In New York, similar legislation is pending.

States are skeptical that AI companies will regulate themselves, said Amina Fazlullah, head of tech policy advocacy for Common Sense Media, an organization that pushes for protections for children online.

“We haven’t seen it work particularly well with social media, specifically for kids,” Fazlullah said.

In December, attorneys general from 39 states and Washington, D.C., sent a letter to Character Technologies and 12 other AI and tech companies—including Anthropic, Meta, Apple, and Microsoft—demanding better protections for children. The letter asked companies to explain how they prevent chatbots from giving harmful advice or impersonating professionals.

These efforts show that states are not waiting for the federal government to act. They are using consumer protection laws, medical licensing rules, and even criminal statutes to hold AI companies accountable.

Practical Takeaways for You

Here is what you can do to protect yourself and your family when using AI chatbots:

• Never trust a chatbot for medical advice. Even if the bot claims to be a doctor, it is not. Use chatbots only for general information, and always verify with a real healthcare provider.
• Check for disclaimers. Many AI platforms include warnings that their characters are not real. But these can be easy to miss, especially for children. Talk to your kids about why they should not believe everything a chatbot says.
• Report suspicious bots. If you see a chatbot pretending to be a doctor, therapist, or other licensed professional, report it to the platform and to your state’s attorney general or medical board.
• Know the signs of a real medical professional. Real doctors have licenses that can be verified through your state’s medical board. They will never diagnose or treat you through a chatbot without a proper exam.
• Talk to your children. Ask your kids if they use chatbots for advice. Explain that chatbots are not people and cannot be trusted with personal health questions.

What Happens Next

The Pennsylvania case will likely take months or years to resolve. If the court rules against Character Technologies, it could set a precedent that AI companies are responsible for what their chatbots say—even if they post disclaimers. That would be a major change from how the law currently treats internet platforms.

If the court sides with the company, it could mean that chatbots are protected speech under federal law, as long as they do not explicitly claim to be real people. Either way, the outcome will affect every AI company that offers conversational bots.

For now, the best advice is simple: treat chatbots like a search engine, not a doctor. They can give you ideas and information, but they cannot replace the judgment, training, and legal responsibility of a licensed healthcare professional. Your health is too important to leave to a machine.