When PHI Meets AI: HIPAA Risks of Pasting Patient Data Into ChatGPT and Other Non-Compliant LLMs
Clinicians are entering patient information into ChatGPT to create therapy plans. But pasting protected health information into non-HIPAA AI models risks fines, leaks, and lost trust.
A doctor, exhausted after a long shift, enters patient details into ChatGPT to help draft the “plan” section of a History and Physical (H&P) note. What feels like a harmless shortcut is actually a HIPAA compliance landmine.
That snippet of Protected Health Information (PHI) just left the hospital’s secure electronic systems and entered an AI model that is not HIPAA-compliant. Depending on the vendor, those details could be logged, retained, or even absorbed into future training runs.
Once a patient’s health information makes its way to an unauthorized chatbot, neither the patient nor the healthcare provider have control over it anymore.
What happens to patient data once it’s on an AI chatbot?
Our tired doc pasted their patient’s medical history, vital signs, and assessment into a general-purpose large language model (LLM) to generate a polished treatment plan.
Unbeknownst to them, the AI vendor’s default setting is to retain prompts for “quality improvement” purposes.
Now, that patient’s name, diagnosis, and clinical details sit on a third-party server. This data is not necessarily protected by the normal digital security provisions HIPAA would offer.
Is ChatGPT HIPAA-compliant?
No, ChatGPT is not compliant with HIPAA by default.
Major AI providers openly state that unless customers sign Business Associate Agreements (BAAs), user inputs may be logged and used for training. Even if the vendor doesn’t actually fine-tune their models on the specific prompt, retention introduces risk: A data breach, subpoena, or shift in vendor policy could expose sensitive patient data.
Takeaway: Drafting or refining a treatment plan in a non-HIPAA compliant AI model is a violation waiting to happen.
Other risks of sharing patient data with an AI model
LLMs are designed to generalize based on their training data, but they are not immune to memorizing data and reproducing it later.
Research has demonstrated it is possible to extract sensitive information from LLMs with a targeted prompt. This makes PHI susceptible to not only regurgitation from the model itself, but potentially to malicious attackers as well.
Once PHI is submitted to a non-HIPAA AI model, there is no reliable way to guarantee it won’t resurface elsewhere.
Are there court cases involving AI HIPAA breaches?
I could not yet find a high-profile case involving a HIPAA breach with ChatGPT or another generative AI application.
When there is an incident (and I truly believe it’s a matter of when), enforcement will likely be aggressive. Health systems have already paid hundreds of thousands for mishandling PHI, and criminal liability can extend up to 10 years for malicious misuse.
In short, regulators will not need precedent to act on an AI-related data breach. If PHI leaks through a chatbot, it will be treated like any other unwanted disclosure.
HIPAA Breach Prevention Plan for AI in Healthcare
The solution to the problem is not to ban AI in healthcare, but to govern it responsibly.
I’ve got three suggestions to keep patient health information safe:
Use HIPPA-compliant AI services. Only use vendors that sign BAAs and guarantee zero data retention. Ensure they use encryption and maintain audit logs.
De-identify PHI rigorously. Remove all identifiers before putting in data; even initials or room numbers can be PHI under HIPAA.
Explore secure, healthcare-specific models. Self-hosted or enterprise-grade AI systems located in your environment provide maximum control, though they require upfront investment.
These steps should be paired with broader AI governance in healthcare: workforce training, vendor due diligence, and regular audits.
Closing thoughts
One careless paste of patient data into ChatGPT can undo years of patient trust (and cost a hospital some serious money).
Generative AI in healthcare will shape the future, but the line between “innovation” and “HIPAA violation” comes down to governance.
The question is not whether PHI will meet AI, but whether leaders will control the terms of that encounter.
Supplement: Glossary of Terms
AI governance in healthcare: The policies and processes that ensure artificial intelligence (AI) systems are developed and deployed responsibly in the healthcare sector. Hospitals can promote patient safety and improve healthcare outcomes by adopting AI governance principles.
Business Associate Agreement (BAA): A contract required under HIPAA between a HIPAA covered entity and a business associate outlining how the business will handle and safeguard PHI.
Large Language Model (LLM): A type of artificial intelligence model that is trained on massive amounts of text data to generate human-like text.
Protected Health Information (PHI): Refers to any individually identifiable health information held or transmitted by a covered entity or its business associate. This includes a an individual’s physical or mental health condition, provision of health care to the individual, or payment for the provision of healthcare to the individual.
Health Insurance Portability and Accountability Act (HIPAA): A U.S. federal law enacted in 1996 which established national standards for handling PHI.
Pro tip: HIPAA is spelled with one P and two As. If you didn’t know, now you know!
Check out my newly-published article here!
Artificial Intelligence in Healthcare: No Longer Optional But Neither Is Patient Safety, found in The American Journal of Healthcare Strategy (Healthcare Strategy Review)
Read how I use AI in my writing here: AI Use Policy
Read how I use analytics to improve my newsletter here: Privacy & Analytics
Hey Ryan,
This is a great article! As healthcare professionals, we see AI as a convenient tool to lessen our workload (which is understandable given the current strain on the system).
But we all forget about the privacy risks until its too late.
My hope is that educated higher ups understand the risks and quickly build policies that protect their staff and their patients instead of pushing for efficiency only. .
Thanks again for writing this!
It’s wild that they even think they can do that.