I want Utah’s AI healthcare pilots to succeed. What does “success” look like?
Why AI healthcare companies cannot elaborate on patient safety before go-live, and why they must once patients are involved.
Utah has authorized two AI systems to autonomously renew prescription refills under defined regulatory limits. Doctronic handles select chronic-disease medications. Legion Health handles select psychiatric maintenance medications.
Most counties in Utah are classified as mental-health provider shortage areas, and primary care wait times can stretch to weeks. The use case for both systems is real, and so is the access problem each is attempting to solve.
They are the first companies of their kind, but they will not be the last. More states will surely implement similar healthcare AI pilots, with more companies gaining approval for clinical AI integration.
But the communications environment surrounding these pilots is structurally incapable of producing one important truth:
Every clinical framework, whether human or AI, produces adverse events at some rate. The irreducible complexity of healthcare cannot be completely mitigated.
The metric for success, then, is whether the system can:
Detect errors before they happen, or soon after they do;
Communicate honestly when errors occur (which, over enough time, is practically a guarantee); and
Make structural changes to the clinical workflow to prevent future errors, rather than claiming it was an “isolated incident.”
Before I go any further, I want to make something very clear. As a healthcare professional, I would like to see all companies in the healthcare AI space be upfront about error margins and patient safety frameworks.
However, the companies who successfully raise capital and get regulatory approval structurally CANNOT be fully transparent about patient safety at the same time.
They will get passed over for a different company which glosses over clinical reality. That’s the argument I make in this article.
Before we begin
I ask that you read on with nuance rather than outrage. Look at my other articles and you’ll see I’m far from a “tech bro” or AI company apologist.
The truth is that we simply cannot assume companies are acting in bad faith in the current startup environment.
Every narrative reaching the public must be calibrated for a specific persuasive task (fund my company; allow us to provide services in your state). Acknowledging that your service can make clinical mistakes to an investor or regulator is practically an unforced error. It invites an increased level of scrutiny, even if it’s the right thing to do.
As for me? I don’t have to craft a narrative for investors or lawmakers. This newsletter talks about keeping patients safe.
So today, you’ll get the Patient Safety Narrative that startups are disincentivized from providing, even if they wanted to.
The standard, since 1999
I said in the introduction that patient safety errors can never be brought to zero, whether you’re the most amazing doctor or the most robust AI system.
That is more than my personal opinion. In fact, the American healthcare system has operated on this principle for over a quarter of a century.
In 1999, the Institute of Medicine (IOM) published To Err Is Human. It changed the working definition of clinical safety in the United States.
The report estimated that medical errors caused tens of thousands of preventable deaths in U.S. hospitals every year. What makes it different is where it put the blame.
The IOM argued these preventable errors were not because clinicians were careless, but because clinical care is performed by humans operating under cognitive load, time pressure, and incomplete information. In other words, mistakes often happen because of the constraints of the healthcare system itself rather than personal failures of the providers.
The goal was to end the assumption that good clinical care meant zero failures. It replaced that assumption with a different standard: errors are inevitable, and what matters is whether the system around the clinician is structured to catch them before they reach the patient.
Since then, the patient safety field has built infrastructure around that standard. Take non-punitive error reporting: if clinicians fear losing their job or license for admitting a mistake, they won’t report it--and the system can’t learn from errors it doesn’t see. The point is system-level learning, not individual blame.
Since AI is becoming a clinical actor, these systems are going to inherit the same irreducible complexity humans grapple with. AI will be more performant than humans in some areas of healthcare and less so in others.
It will make mistakes I could predict today, and also probably err in ways no one could have anticipated. That’s why AI companies need to be transparent about their mistakes and commit to improving their processes rather than framing incidents as isolated.
Once real people’s lives are on the line, investor speak doesn’t cut it any longer.
But why can’t the companies deeply discuss patient safety right now?
It seems counterintuitive that healthcare AI companies cannot be upfront about how they will apply the field’s actual working standards. It’s frustrating to me. I’m sure it’s also frustrating to the companies themselves.
The reality as it stands now, though, is that every narrative from the company must be calibrated for a different persuasive task, and each task penalizes honesty about the risk of errors.
I explained this concept in depth in an earlier piece, but here’s a brief summary of why being candid about patient safety works against them.
Investor messaging
If my hypothetical startup is upfront about realistic error margins, I might be competing with other companies who claim their service matches physician recommendations over 99% of the time.
The company who is candid probably loses the fundraising round to the companies who are not.
Clinician messaging
Healthcare professionals operating inside these workflows have their licenses on the line.
Emphasizing AI error rates to clinicians will guarantee that they will override your suggestions more, not adopt them. While this might be better for patient safety measured holistically, it hurts a critical metric that regulators are holding their pilots to: AI-physician concordance rates.
Patient messaging
Companies are targeting their marketing efforts to patients who are frustrated with their current care. They may not be receiving adequate treatment right now; that’s why they want to switch.
Converting those patients to your service is infinitely more difficult if you mention the risk for harm in your ads. It’s a better play to put those things on your website’s FAQ.
The drawback is patients may only read that after they’ve signed up for the service and are coming to realize their specific needs cannot be met.
Regulator messaging
Getting your service approved by regulators is a binary process. Your company either gets the rubber stamp to proceed or it doesn’t.
Volunteering precise error expectations forecloses ambiguity the landscape often currently permits, and may trigger reporting requirements the company hasn’t yet built infrastructure for. This is inconvenient and probably costly.
The pattern holds across every other channel the company operates in: health system contracts, social media posts, coalition letters, trade-press interviews.
Making each locally correct decision for your company’s success currently necessitates forgoing honest discussions about patient safety. Yes, necessitates it.
My message to healthcare AI startups
Since I want companies to actually listen to me about patient safety concerns, telling them to be more transparent before go-live isn’t going to work. We’ve established that they cannot listen and still succeed, even if they wanted to. (I’ll save that advocacy for the legislators.)
Instead, this is my message for Doctronic, Legion Health, and every future company who follows in their footsteps:
Right now, your company must gloss over the specifics of how you’ll handle patient safety events to get your funding and regulatory approval. I understand and acknowledge that, even if I don’t like that it has to be this way.
But once you’re dealing with the health of someone’s grandparent, or sibling, or friend, your narrative register must shift dramatically.
Once you start taking care of patients, you must use the language and frameworks of patient safety like the rest of us.
It is not acceptable to tell investors or lawmakers that a patient safety event which hit national news was an “isolated incident,” or the fault of the healthcare provider who is already working under other extreme pressures.
Our healthcare system left that rhetoric behind in the 20th century.
These are the three things you must do for your service to be accepted by healthcare providers, your real end users:
Detect errors before they reach patients. Have a system for the ones that slip through.
Be honest about patient safety events. Publicly.
Make proactive structural changes to your workflows in response.
I want our patients to be safe, so I truly wish for your resounding success.
Welcome to the trenches.
Ryan Sears, Philly’s AI Pharmacist
Effort disclosure: This is a commentary and framework synthesis. It builds on primary investigation of the Utah AI healthcare pilot program agreements covered in earlier pieces, and on the “13 Narratives of Healthcare AI Companies” framework I previously authored. I did not request public records or perform direct outreach for this piece.
Read how I use AI in my writing here: AI Use Policy
Read how I use analytics to improve my newsletter here: Privacy & Analytics
Well stated and thought through (like in your earlier piece). As someone also working in this area (for my day job), I do think this will and must change. There is yet to be concerted attention to a crisis to force the bioethical discussions we need to have around where responsibility and accountability lie with AI being used as augmentative and, increasingly, substitution for medical professionals’ judgment. Look at the throughput and I don’t think we can ignore the latter or the push to get to substitution more broadly.
That crisis, accident, etc. will happen. Then the court cases, then the frameworks that are out there will start to be applied and the communication challenges will shift. I do hope those ethical questions can become the norm before all of that, but we are not in that environment with the stressors on the healthcare system as a whole and the promises, inflated or actual, at hand.
Great thinking and writing!
Ryan. Thanks for this write up. Clarifying question - I don't see a position from you on whether patients should be able to opt out. Medicine can be so personal and AI pilots do tend to raise skepticism from patients. How does patient opt out fit?