Most counties in Utah are classified as mental-health provider shortage areas, and primary care wait times can stretch to weeks. The use case for both systems is real, and so is the access problem each is attempting to solve.

They are the first companies of their kind, but they will not be the last. More states will surely implement similar healthcare AI pilots, with more companies gaining approval for clinical AI integration.

But the communications environment surrounding these pilots is structurally incapable of producing one important truth:

Every clinical framework, whether human or AI, produces adverse events at some rate. The irreducible complexity of healthcare cannot be completely mitigated.

The metric for success, then, is whether the system can:

• Detect errors before they happen, or soon after they do;

• Communicate honestly when errors occur (which, over enough time, is practically a guarantee); and

• Make structural changes to the clinical workflow to prevent future errors, rather than claiming it was an “isolated incident.”

Before I go any further, I want to make something very clear. As a healthcare professional, I would like to see all companies in the healthcare AI space be upfront about error margins and patient safety frameworks.

However, the companies who successfully raise capital and get regulatory approval structurally CANNOT be fully transparent about patient safety at the same time.

They will get passed over for a different company which glosses over clinical reality. That’s the argument I make in this article.

Before we begin

I ask that you read on with nuance rather than outrage. Look at my other articles and you’ll see I’m far from a “tech bro” or AI company apologist.

The truth is that we simply cannot assume companies are acting in bad faith in the current startup environment.

Every narrative reaching the public must be calibrated for a specific persuasive task (fund my company; allow us to provide services in your state). Acknowledging that your service can make clinical mistakes to an investor or regulator is practically an unforced error. It invites an increased level of scrutiny, even if it’s the right thing to do.

As for me? I don’t have to craft a narrative for investors or lawmakers. This newsletter talks about keeping patients safe.

So today, you’ll get the Patient Safety Narrative that startups are disincentivized from providing, even if they wanted to.

The standard, since 1999

I said in the introduction that patient safety errors can never be brought to zero, whether you’re the most amazing doctor or the most robust AI system.

That is more than my personal opinion. In fact, the American healthcare system has operated on this principle for over a quarter of a century.

In 1999, the Institute of Medicine (IOM) published To Err Is Human. It changed the working definition of clinical safety in the United States.

The report estimated that medical errors caused tens of thousands of preventable deaths in U.S. hospitals every year. What makes it different is where it put the blame.

The IOM argued these preventable errors were not because clinicians were careless, but because clinical care is performed by humans operating under cognitive load, time pressure, and incomplete information. In other words, mistakes often happen because of the constraints of the healthcare system itself rather than personal failures of the providers.

The goal was to end the assumption that good clinical care meant zero failures. It replaced that assumption with a different standard: errors are inevitable, and what matters is whether the system around the clinician is structured to catch them before they reach the patient.

Since then, the patient safety field has built infrastructure around that standard. Take non-punitive error reporting: if clinicians fear losing their job or license for admitting a mistake, they won’t report it--and the system can’t learn from errors it doesn’t see. The point is system-level learning, not individual blame.

Since AI is becoming a clinical actor, these systems are going to inherit the same irreducible complexity humans grapple with. AI will be more performant than humans in some areas of healthcare and less so in others.

It will make mistakes I could predict today, and also probably err in ways no one could have anticipated. That’s why AI companies need to be transparent about their mistakes and commit to improving their processes rather than framing incidents as isolated.

Once real people’s lives are on the line, investor speak doesn’t cut it any longer.

But why can’t the companies deeply discuss patient safety right now?

It seems counterintuitive that healthcare AI companies cannot be upfront about how they will apply the field’s actual working standards. It’s frustrating to me. I’m sure it’s also frustrating to the companies themselves.

The reality as it stands now, though, is that every narrative from the company must be calibrated for a different persuasive task, and each task penalizes honesty about the risk of errors.

I explained this concept in depth in an earlier piece, but here’s a brief summary of why being candid about patient safety works against them.

Investor messaging

If my hypothetical startup is upfront about realistic error margins, I might be competing with other companies who claim their service matches physician recommendations over 99% of the time.

The company who is candid probably loses the fundraising round to the companies who are not.

Clinician messaging

Healthcare professionals operating inside these workflows have their licenses on the line.

Emphasizing AI error rates to clinicians will guarantee that they will override your suggestions more, not adopt them. While this might be better for patient safety measured holistically, it hurts a critical metric that regulators are holding their pilots to: AI-physician concordance rates.

Patient messaging

Companies are targeting their marketing efforts to patients who are frustrated with their current care. They may not be receiving adequate treatment right now; that’s why they want to switch.

Converting those patients to your service is infinitely more difficult if you mention the risk for harm in your ads. It’s a better play to put those things on your website’s FAQ.

The drawback is patients may only read that after they’ve signed up for the service and are coming to realize their specific needs cannot be met.

Regulator messaging

Getting your service approved by regulators is a binary process. Your company either gets the rubber stamp to proceed or it doesn’t.

Volunteering precise error expectations forecloses ambiguity the landscape often currently permits, and may trigger reporting requirements the company hasn’t yet built infrastructure for. This is inconvenient and probably costly.

The pattern holds across every other channel the company operates in: health system contracts, social media posts, coalition letters, trade-press interviews.

Making each locally correct decision for your company’s success currently necessitates forgoing honest discussions about patient safety. Yes, necessitates it.

My message to healthcare AI startups

Since I want companies to actually listen to me about patient safety concerns, telling them to be more transparent before go-live isn’t going to work. We’ve established that they cannot listen and still succeed, even if they wanted to. (I’ll save that advocacy for the legislators.)

Instead, this is my message for Doctronic, Legion Health, and every future company who follows in their footsteps:

Right now, your company must gloss over the specifics of how you’ll handle patient safety events to get your funding and regulatory approval. I understand and acknowledge that, even if I don’t like that it has to be this way.

But once you’re dealing with the health of someone’s grandparent, or sibling, or friend, your narrative register must shift dramatically.

Once you start taking care of patients, you must use the language and frameworks of patient safety like the rest of us.

It is not acceptable to tell investors or lawmakers that a patient safety event which hit national news was an “isolated incident,” or the fault of the healthcare provider who is already working under other extreme pressures.

Our healthcare system left that rhetoric behind in the 20th century.

These are the three things you must do for your service to be accepted by healthcare providers, your real end users:

1. Detect errors before they reach patients. Have a system for the ones that slip through.

2. Be honest about patient safety events. Publicly.

3. Make proactive structural changes to your workflows in response.

I want our patients to be safe, so I truly wish for your resounding success.

Welcome to the trenches.

Ryan Sears, Philly’s AI Pharmacist

Subscribe now

Effort disclosure: This is a commentary and framework synthesis. It builds on primary investigation of the Utah AI healthcare pilot program agreements covered in earlier pieces, and on the “13 Narratives of Healthcare AI Companies” framework I previously authored. I did not request public records or perform direct outreach for this piece.

Read how I use AI in my writing here: AI Use Policy

Read how I use analytics to improve my newsletter here: Privacy & Analytics

Within a single quarter, a company may need to persuade investors that its market is enormous, and at the same time tell regulators its scope is narrow. They might tell health system executives that the product will transform hospital workflows while trying to convince clinicians that they will retain their autonomy in clinical practice.

Each of these audiences expects a different kind of communication and evaluates the company on different criteria. Because of this, the company must calibrate its wording of concepts differently for every group it communicates with.

This is not, by itself, a problem. Different audiences need different things. For example:

• A regulator needs precision about what the product is claiming to do.

• An investor wants to see that the product can scale.

• A patient needs to know whether the product can help with their health goals.

The question I’ll be exploring in this article is what happens when the language a company uses with two different groups might contradict each other.

When contradictions appear in the register of healthcare AI companies, they are often not random. Some of them are unavoidable given how many groups the company has to communicate with, and some are genuine discrepancies that warrant further attention.

This article is intended to be a general framework that you can apply to any healthcare AI company you choose. It is not written about any company in particular and that’s on purpose.

Key takeaway: Reading only one of the company’s narratives will not give you the full picture. Comparing all the narratives next to each other is the only way to understand what’s happening.

Subscribe now

Defining the 13 narratives

Each narrative below is presented in a consistent labeled format:

1. Where the narrative appears.

2. Who the audience is.

3. What the audience expects.

4. What the company is trying to persuade the audience of.

5. How the narrative is constructed.

6. What legitimate purpose the narrative serves.

7. The structural cost when comparing every narrative together (in other words, where potential discrepancies in the narrative may occur).

After I lay out all the narratives, I will describe how to reconcile all the narratives for different groups such as patients, clinicians, and healthcare executives.

#1: The Capital Narrative

Appears in: Pitch decks, investor updates, board materials, fundraising press releases, and the public pages where investors describe the companies they fund.

Audience: Current investors, prospective investors, and the financial press covering the sector.

Expectations: Ambition calibrated to the size of the opportunity the company is asking them to fund.

Persuasion target: That the company is worth funding at the proposed valuation, that the market opportunity is large enough to justify the projected returns, and that the current team and technology are positioned to capture it.

Narrative strategy: The company’s market is described at the largest defensible scale. Growth rates are presented as accelerating. The technology is described as a category-defining advance over existing alternatives. Competitors are framed as incumbents whose business models are about to be disrupted. Future capabilities are presented alongside current capabilities with minimal distinction between them. Unit economics are described in terms of projected efficiency rather than current performance.

Purpose: Investors evaluate companies on potential rather than present-state performance. The pitch register exists to communicate that potential efficiently.

Structural cost: Investors expect this register and read it with the appropriate level of skepticism regarding the claims. However, this language can make its way to more than just investor groups. Other audiences may take every claim at face value.

#2: The Health System Narrative

Appears in: Vendor proposals, Request for Proposal (RFP) responses, executive briefings, board presentations, strategic partnership announcements, contract documents, and the materials directed at health system executives evaluating whether to deploy the product at scale.

Audience: Hospital and health system executives, chief medical officers, chief information officers, chief financial officers, procurement teams, and the strategic decision-makers who sign contracts deploying healthcare AI products across their organizations.

Expectations: Business case justification, integration feasibility, financial modeling, risk allocation terms, and a strategic positioning story the health system can carry to its board of directors.

Persuasion target: That deploying this product advances the system’s strategic and financial position, that the integration and operational risks are manageable, that the regulatory and liability exposures are acceptable, and that the deployment will produce measurable returns that justify the contract.

Narrative strategy: The product is framed as a strategic investment rather than as a clinical tool. Cost reduction is quantified. Throughput improvements are projected.

Competitive positioning against peer systems is emphasized. Integration with existing systems is described as straightforward.

Liability and regulatory risks are described as already addressed through the company’s compliance infrastructure.

Clinician adoption is described as achievable through the company’s implementation support. The product’s role within the system’s workflow is described in terms that align with the system’s existing strategic narratives about innovation, efficiency, or access.

Purpose: Health systems are large, complex organizations that evaluate vendor products on business criteria. The system narrative provides the executive-facing case that contracts depend on.

Structural cost: The system narrative and the clinician narrative often describe the same product to two audiences within the same organization, and the descriptions can be materially different.

The health system narrative may describe the product as reducing clinician labor; the clinician narrative may describe it as supporting clinician judgment.

The health system narrative may project throughput gains that depend on clinician workflow changes which the clinician narrative might later soften.

The translation layer between system-level decision-making and clinician-level implementation is where these contradictions are typically resolved or glossed over. The clinicians who will actually be using the product may or may not be invited to contribute in these discussions.

#3: The Patient Marketing Narrative

Appears in: Advertising, paid search results, social media targeting, app store listings, and consumer-facing pages designed to convert prospects into signups.

Audience: Regular people experiencing the conditions the product addresses, often during moments of frustration with the current healthcare (or lack thereof) they are receiving.

Expectations: Information about whether this product can help them.

Persuasion target: That the product is worth signing up for, that it can address the patient’s healthcare needs more easily or affordably than alternatives, and that the friction of switching from existing care is justified by the benefits.

Narrative strategy: Accessibility, affordability, and convenience are emphasized. Wait times are described as short. Costs are presented in terms of monthly subscription pricing rather than total cost of care.

The conditions the product addresses are described broadly. Outcomes are described in terms of patient satisfaction or symptom improvement.

Limitations, exclusions, and clinical constraints are usually not laid out in full.

Purpose: Patients searching for healthcare need to find and evaluate options. The marketing register communicates availability efficiently.

Structural cost: The marketing register describes the product as if all interested patients are appropriate candidates.

The product’s actual eligibility criteria, scope limitations, and clinical exclusions are typically not disclosed at this stage.

Patients form expectations from the marketing register that the product’s operational parameters may not always meet.

#4: The Patient FAQ Narrative

Appears in: Product FAQs, terms of service, in-app onboarding flows, and post-signup communications.

Audience: Patients who have already engaged with the marketing register and are now encountering the product’s constraints.

Expectations: Clarification of how the product really works.

Persuasion target: That the constraints encountered post-signup are appropriate clinical safety measures rather than a downgrade from what the marketing implied, and that the patient should continue with the product despite the narrower-than-expected scope.

Narrative strategy: The product’s scope is described in narrower terms than the marketing register suggested. Specific conditions are excluded. Specific medications are excluded.

Provider supervision is described as a feature rather than a constraint.

Edge cases--what happens if symptoms worsen, if the patient is in crisis, if the patient needs medication not in the formulary--are addressed with referral language.

The patient’s responsibilities when using the product are spelled out.

Purpose: Patients need to understand the product’s actual parameters before relying on it for their care needs.

Structural cost: The FAQ register often reaches patients only after they have signed up. The marketing register that brought them in may have described a wider product.

Patients may end up paying for a product whose true scope does not match what attracted them to the product initially.

#5: The Regulatory Narrative

Appears in: Binding regulatory documents, state agency filings, compliance reports, communications with licensure boards, and federal regulatory submissions.

Audience: State and federal regulators with statutory authority over the product’s approval, continuation, and/or expansion.

Expectations: Precise, narrowly defined claims about scope, capability, and operational parameters.

Persuasion target: That the product fits within authorized regulatory scope, that the company’s operational practices match the regulatory representations, and that ongoing oversight is satisfied by the company’s reporting and compliance infrastructure.

Narrative strategy: The product’s scope is described with specificity. Eligibility criteria are itemized. Excluded conditions and medications are listed. Reporting requirements are acknowledged. Phased implementation timelines are committed to.

The product’s role is described in terms that match the regulatory framework’s existing categories.

Where the framework permits ambiguity, the company takes advantage; where the framework requires specificity, the company supplies it.

Purpose: Regulators need precise descriptions to evaluate whether the product fits within authorized scope.

Structural cost: The regulatory narrative is often the only narrative legally binding on the company, and it is also the narrative that almost no one outside the regulatory audience reads.

Patients, recruits, and investors form their understanding of the product from registers the company might not be legally bound to.

In other words, the regulatory narrative governs what the product can actually do, and the other narratives shape what people think it can do.

#6: The Company Recruit Narrative

Appears in: Company job postings, recruiter outreach, careers pages, engineering-focused company blogs, materials directed at clinical hires, and the public-facing communications designed to attract talent across operational and clinical roles.

Audience: Experienced operational hires (engineers, product, ops, sales, marketing) and licensed clinical hires (physicians, advanced practice providers, pharmacists, therapists) the company wants to recruit away from existing employers.

Expectations: Persuasion that the opportunity justifies leaving a stable role for a younger company.

Operational recruits expect ambition and equity upside.

Clinical recruits expect mission alignment and assurance that the company respects professional judgment.

Persuasion target: That the opportunity offers career and financial upside that exceeds the recruit’s current trajectory, and for clinical recruits, that the company’s clinical structure preserves the professional standards the recruit would expect from a traditional practice setting.

Narrative strategy: For operational recruits, the company is described as building something foundational. Equity is framed in terms of upside. The work is described as hard problems experienced operators are best positioned to solve. Career trajectory is framed as accelerated by the company’s stage.

For clinical recruits, the AI is described as augmenting rather than replacing clinical judgment. Licensed clinicians are described as the responsible decision-makers. Compensation, autonomy, and professional development are highlighted alongside mission alignment.

The two recruit sub-registers often run in parallel, sometimes even in the same materials.

Purpose: Companies need to attract talent from established employers. Senior operational hires evaluate whether the opportunity justifies the move. Licensed clinicians evaluate employment against professional standards as well as economic ones.

Structural cost: The two recruit sub-registers might contradict each other.

The operational recruit register may describe the company as shifting clinical labor to AI.

The clinical recruit register describes the AI as a tool supporting licensed clinicians.

The same materials can carry both messages because they reach different readers. However, it is just as possible that in each sub-register may not see the language directed at the other.

#7: The Clinician Narrative

Appears in: Clinical workflow tools, supervising-physician guidance documents, standing orders and protocols, internal training materials, performance dashboards, supervising-physician meetings, and the ongoing operational communications directed at licensed clinicians who deliver supervised care through the product.

Audience: Physicians, nurse practitioners, pharmacists, therapists, and other licensed clinicians currently operating within the company’s supervised-care framework.

Expectations: Guidance that justifies adopting the product’s workflow and operating within the company’s defined scope of professional judgment.

Persuasion target: That adopting the product’s workflow is clinically sound, that AI-generated recommendations are reliable enough to act on with the level of review the workflow asks for, that operating within the company’s defined scope of professional judgment is consistent with the clinician’s professional standards and patient welfare, and that the supervised-care framework genuinely protects both patients and the clinician’s professional license.

This is among the most demanding persuasive tasks in the entire communications environment because clinical training often cultivates resistance to workflow changes that don’t clearly serve individual patients, and because the consequences of misplaced trust fall both on the clinician’s license and on patient outcomes simultaneously.

Narrative strategy: Clinical guidance frames the workflow as protecting patient safety while enabling efficiency at scale.

Protocols are presented as evidence-based and as expressions of clinical best practice.

AI-generated recommendations are described as decision support that the clinician evaluates and approves.

Performance metrics are framed as measuring quality of care, not just throughput.

Workflow constraints are framed as scope discipline rather than as restrictions on professional judgment.

Supervising physicians are positioned as clinical leaders whose authority shapes the product’s safety.

The cumulative register asks clinicians to trust the company’s clinical infrastructure enough to operate within it.

Purpose: Clinicians have established professional habits and standards that the company’s workflow asks them to adapt. The clinician narrative exists to persuade clinicians that the adaptation is clinically sound, professionally appropriate, and aligned with patient welfare.

Structural cost: The clinician narrative determines whether the supervised-care framework described to regulators actually operates as described.

If the persuasion succeeds in producing substantive clinical review, the framework functions. If the persuasion succeeds in producing routine sign-off on AI-generated outputs, the framework exists on paper but not in practice.

The company has significant latitude in setting expectations through training design, workflow architecture, and performance metrics. The clinician narrative is the lever that determines which version of supervised care actually operates.

#8: The Evidence Narrative

Appears in: Peer-reviewed journal publications, medical society presentations, preprint repositories, company-authored white papers, and academic conference materials.

Audience: Clinicians, researchers, and policy analysts who evaluate products through formal evidence frameworks.

Expectations: Claims about clinical performance supported by methodologically sound studies.

Persuasion target: That the product’s clinical performance is supported by formal evidence sufficient to justify clinical adoption, regulatory authorization, and continued use within the medical literature’s existing frameworks.

Narrative strategy: The company presents data on its product’s performance, often in collaboration with academic partners.

Study designs emphasize favorable comparisons. Endpoints are selected to demonstrate the product’s value within its tested scope. Limitations are acknowledged in the discussion sections in conventional academic language.

The product’s role is described in terms that align with existing medical literature frameworks.

Purpose: Clinical products require evidence to support their use. The evidence-positioning register communicates that evidence in formats clinical and research audiences trust.

Structural cost: The underlying research is often designed by the entity whose product is being evaluated.

Preprints can travel into other registers with academic credibility before they are fully peer reviewed.

The capital narrative may cite the evidence narrative to support claims which evidence itself does not support.

#9: The Founder Personal Narrative

Appears in: AI healthcare company founders’ personal social media accounts, podcast appearances, and informal public communications outside of the company’s “official” channels.

Audience: Fellow founders, investors, and industry watchers.

Expectations: Directness and ambition beyond what formal company communications may normally carry.

Persuasion target: That the founder personally believes in the company’s destination strongly enough to stake their reputation on it, and that this conviction is itself evidence of the company’s prospects.

Narrative strategy: The product’s technology is described as transformative. The company’s success is treated as a guarantee.

Existing industry constraints--such as current regulatory frameworks, established care models, or professional norms--might be framed as merely temporary obstacles. Timelines compress to fit the founder’s vision and ambitions.

Purpose: Founders need to project the conviction that attracts talent and capital, and personal communications are where that conviction registers most authentically.

Structural cost (at least in theory; I am not a lawyer, and this is not legal advice): Personal communications about company business may not always be legally separate from company communications.

In the past, regulators have treated founder social media as corporate disclosure; courts may find such communications discoverable in litigation involving the company.

In other words, personal channels can still be part of the company’s public record.

#10: The Coalition Narrative

Appears in: Materials co-produced with trade associations, industry groups, academic partners, political action committees, and aligned policy organizations.

Audience: Legislators, regulators, professional society members, and the broader audiences those organizations reach.

Expectations: Messaging that reflects the coalition’s institutional positioning rather than any single company’s marketing.

Persuasion target: That the regulatory environment should accommodate the industry’s preferred operating conditions, framed as serving shared sector interests rather than any individual company’s commercial interests.

Narrative strategy: The company is one of several signatories to letters, position papers, comment submissions, or coordinated communications.

Industry-wide framings emphasize the need for regulatory flexibility.

Recommendations are framed as serving shared sector interests. Individual companies and products are not usually emphasized in this context.

The coalition register provides the company with policy influence that does not require it to make claims under its own name.

Purpose: Industries need collective representation in policy conversations. Coalitions provide that representation.

Structural cost: Coalition communications often shape regulatory outcomes that affect specific companies, but the public association with any particular company is not always clear.

A company whose marketing register makes ambitious claims can sign coalition letters whose register is measured and policy-oriented.

Coalition communications also create channels through which industry framings reach trade press and academic outlets as if they originated independently of the companies that helped produce them.

#11: The Friendly Press Narrative

Appears in: Interviews with trade press, business press coverage of the company’s domain, and articles by journalists with established relationships with the company’s communications team.

Audience: Industry watchers, business decision-makers, and general readers interested in the domain.

Expectations: Measured, contextualized coverage rather than promotional content.

Persuasion target: That the company is more thoughtful and prudent than its capital register would suggest, and that the company is engaged with the domain’s complexity rather than dismissive of it.

Narrative strategy: The founder or executive speaks with thoughtful prudence. The company’s mission is framed in terms of long-standing problems the technology is helping to address.

Risks are acknowledged in general terms. Timelines are not committed to specifically. Regulatory engagement is described as collaborative.

Purpose: Trade press needs access to executive perspectives to cover the domain credibly. Companies need a register that is calibrated for trade press conventions.

Structural cost: The same company executive who speaks in this register to trade press also speaks in the capital narrative to investors and the personal narrative on social media.

A reader who only read the trade press coverage may not be aware of the other language the company has used to describe their product.

#12: The Hostile Press Narrative

Appears in: Company responses to independent journalists, regulatory critics, or investigative outlets that challenge the company’s other narratives.

Audience: The readership of the critical coverage, the journalists themselves, and the broader public observing the controversy.

Expectations: Substantive response from the company regarding critiques.

Persuasion target: That the criticisms are misplaced or have been addressed, that the company’s oversight infrastructure is adequate as it currently exists, and that operations should continue without public or regulatory backlash.

Narrative strategy: The company emphasizes its commitment to safety, oversight, and procedural compliance.

Substantive claims from earlier registers are not retracted but are reframed in narrower terms.

The company expresses welcome of regulatory scrutiny while disputing characterizations of its conduct.

Responses will focus on procedural correctness--what was filed when, what reporting was completed, what oversight applies to them--rather than on the substantive claims about whether the product’s actual operation matches its public descriptions.

Direct questions are often redirected to other venues.

Purpose: Companies facing public criticism need a register that addresses criticism without conceding more than the facts require.

Structural cost: The defensive register often reveals what the company actually believes its product is, when forced to be specific.

The procedural emphasis shows what the company is legally bound to. The reframed claims show what the company can substantiate when pressed.

Readers who compare this narrative to earlier ones can see which claims survived scrutiny and which did not.

#13: The Crisis Response Narrative

Appears in: Company communications during a specific incident such as a patient safety event, a regulatory enforcement action, an investigative news story, a lawsuit, or other event that demands immediate public response.

Audience: Parties affected by the incident, journalists covering it, regulators evaluating it, and the broader public observing how the company handles adverse events.

Expectations: Accountability, transparency, and commitment to corrective action.

Persuasion target: That the incident is isolated rather than systemic, that the company’s existing infrastructure is adequate to address it, and that ongoing operation should continue without the regulatory action or public reaction that the incident might otherwise produce.

We can think of this as essentially a more acute and possibly more viral form of the hostile press narrative.

Narrative strategy: The company expresses concern, commits to thorough review, and emphasizes its existing safety infrastructure.

Specific incidents are characterized as isolated or already addressed by existing protocols.

Substantive claims about cause, responsibility, or systemic factors are typically deferred pending review. Legal counsel shapes the register significantly.

Statements are crafted to satisfy multiple audiences simultaneously without committing to specifics that could be used against the company later.

Purpose: Companies facing adverse events need a register that acknowledges seriousness while preserving legal and operational flexibility.

Structural cost: The crisis register reveals what the company defaults to when forced to be specific. The procedural emphasis shows what the company is legally required to do. The deferral of substantive claims shows what the company is unwilling to commit to.

Readers who compare this register to other narratives can see which prior claims the company will defend under pressure and which it will not.

Examples of potential narrative contradiction scenarios

When read side-by-side, the thirteen narratives together reveal something no single narrative would.

The pattern that arises is not random.

When a company needs to persuade a specific audience of something specific badly enough, the calibrated language for that audience is more likely to diverge from the more cautiously calibrated messages which the company typically delivers.

Contradictions across narratives, therefore, are typically produced by the asymmetric urgency of different persuasive tasks.

I want to highlight three situations where the pattern can be most clearly seen:

The first sits inside a single healthcare system.

The executives who sign the contract and the clinicians who deliver care through the product belong to the same organization, and they often receive descriptions of the product that read as if they describe different products entirely.

The executive-facing description emphasizes strategic returns, throughput gains, and managed risk.

The clinician-facing description emphasizes decision support, supervised judgment, and patient safety. The gap is not always reconcilable. It is most likely to surface at the workflow level, only after the contract has already been signed.

The second pairing involves the same patient at two points in time: before and after signing up for the AI healthcare product.

The marketing narrative reaches a person searching for care and persuades them that the product is broadly accessible, affordable, and convenient.

The FAQ narrative reaches the same person after signup and clarifies that the operational scope is narrower than the marketing implied.

The narrowing arrives once the friction of switching out is higher than the friction of staying in.

The third pairing involves the company’s own language when speaking to investors and regulators.

The capital narrative describes a market the company will ambitiously reshape; the regulatory narrative describes the narrow scope the company will operate inside of.

Across all three examples, the same logic operates: the register which necessitates the most persuasive urgency produces the strongest claims, and the strongest claim is most likely to move away from from what other registers must also say.

Narrative reconciliation

This section will discuss concrete steps to reconcile some of the discrepancies that may arise in a healthcare AI company’s communications.

It does not make judgments on which are structurally inevitable and which are true discrepancies worth further scrutiny. That is a process readers must make themselves on a case-by-case basis.

Reconciling the health system-clinician narrative gap

For executives evaluating a contract: before signing, request three documents the sales process may not always surface.

First, the draft clinician training materials. Look specifically for the time-per-encounter targets, the supervising physician review requirements, the override and escalation pathways, and the performance metrics clinicians will be measured on.

Second, the actual workflow integration plan from the implementation team (not just the executive summary).

Third, the override rate data from peer deployments, if the vendor has it.

Once you have these, compare the time the workflow allots for clinical judgment against the time the throughput projections in the business case require.

If the two do not reconcile in the arithmetic, the throughput commitments cannot be met without the workflow being faster than the protocol describes.

For clinicians operating inside an AI-assisted workflow: document the specific review steps the protocol asks of you.

Compare them to the supervised-care framework named in your training materials, any regulatory disclosure, and the company’s public-facing description of its clinical model.

Track your own override rates over time. If your override rate trends toward zero over the first six months, it’s possible that the framework exists on paper but not in practice.

As with anything else in healthcare, it’s important to document findings and escalate serious patient safety concerns. I won’t get into specifics about what can or should happen next in a general framework piece like this.

Reconciling the patient marketing and patient FAQ narrative gap

Before signing up for any healthcare AI product, find the FAQ. It is usually linked from the footer of the marketing site or in the app store description.

Read four specific sections:

1. The list of excluded health conditions. Do the exclusions apply to you?

2. The list of medications in the program’s formulary (a formulary is list of approved medications). Are your medication(s) on there?

3. The referral language describing what happens when the product cannot help.

4. The full cost structure beyond the monthly subscription.

Compare these specifics to the marketing copy. The marketing copy implies coverage; the FAQ specifies non-coverage.

If the condition you actually have appears in the exclusion list, the product is not for you regardless of what the marketing implied.

For clinicians, family members, and patient advocates: when a patient mentions using a particular product, ask them what they thought they were getting and what they are actually getting now.

The gap between the two is often where care has been narrowed without the patient’s full awareness.

Reconciling the capital narrative and regulatory narrative gap

Find the regulatory filings before reading the pitch.

Look at agency websites, FDA submissions, Open Payments, records from the state board of medicine/nursing/pharmacy, and any agency where the company has applied for or received authorization to operate.

Look specifically for phased implementation commitments, reporting requirements, scope limitations on patient populations, conditions or medications the company can address, and any compliance findings or corrective actions. These documents define what the company can legally do.

Then read the pitch deck or investor update. Compare the market sizing, growth projections, and expansion plans against the regulatory scope you just confirmed.

If the projected market depends on the company operating in conditions, geographies, or medications the regulator has not authorized, the company is either planning to expand its authorization, planning to operate beyond it, or projecting beyond what it can actually do.

If this is unclear, ask the company directly which one is true. The answer is rarely volunteered.

For investors specifically: ask the company directly, in writing, for a reconciliation of total addressable market against current authorized scope. The reconciliation may be reasonable. The absence of one is itself a finding.

For regulators and policy watchers: investor materials might display what a company plans to do beyond current authorization. Pitch decks speak to a different audience than regulatory filings, and any difference between the two is the gap.

Reconciling everything

For readers without a position inside any single register, the analytical move requires triangulation.

First, watch the founders, not just the company. Founder personal communications may not always have the same level of editorial filtration applied to capital, regulatory, and friendly press narratives.

Subscribe to founder feeds on the platforms they use. Search podcast platforms for their interviews. Note specific claims about the product, the market, the regulatory environment, and the company’s intentions.

When formal company communications later contradict those claims, the founder’s earlier register often shows what the company believed it was building before the editorial filtration was applied.

Second, archive crisis responses while they are fresh. This can be done on sites like The Wayback Machine and archive.today.

When a company is named in an investigative story, an enforcement action, or a lawsuit, save the company’s specific public statements (press releases, comments to journalists, court filings if accessible, regulatory responses if public).

The procedural emphasis in crisis language, references to oversight infrastructure, timely filings, and welcomed regulatory scrutiny, shows what the company is legally bound to do.

The reframing of earlier claims in narrower terms shows what the company can substantiate when pressed. Both registers are usually preserved best by people who saved them in real time.

Third, trace evidence claims to their underlying studies. When the capital register or a friendly press piece cites peer-reviewed work, find the study. Note the sample size, the comparator, the primary and secondary endpoints, and the inclusion and exclusion criteria. Compare what the study claims to what the company says the study supports. The headline claim and the underlying claim are often different sizes, and the difference is usually legible only at the study level.

Fourth, read coalition communications with the signatories visible. Trade group letters, position papers, and policy comment submissions often shape regulatory outcomes affecting specific companies.

The coalition’s framing reaches journalists and legislators as if it originated independently. Tracing the signatories and the funding behind the coalition’s coordination can surface the individual companies’ interests behind the coalition framing.

Keep a discrepancy log: specific quotes, specific dates, specific registers. Patterns emerge over time, and the log becomes the evidence base for any subsequent question worth asking on the record.

The framework above is descriptive, not prescriptive. It does not say which contradictions are unethical and which are unavoidable. It does not say which companies are operating in good faith and which are not.

It says that the structural conditions of the communications environment make certain kinds of cross-narrative contradictions predictable.

The most useful contribution that you, the reader, can make to healthcare AI governance is asking sharper questions, about specific registers, to the people whose answers actually matter.

Read how I use AI in my writing here: AI Use Policy

Read how I use analytics to improve my newsletter here: Privacy & Analytics

This is the second article of the series on Legion Health.

If you missed the first one, read it here.

Quick recap for new readers

Earlier this month, Utah approved a 12-month regulatory mitigation agreement with Legion Health, a San Francisco AI startup, to allow AI-facilitated renewals of a narrow set of psychiatric prescriptions that a human clinician had already prescribed. The pilot is now live, and leadership has publicly described nationwide ambitions.

The decision landed in a state where 99% of the geography is a mental health professional shortage area and roughly half a million Utahns lack adequate behavioral healthcare access.

My first article asked readers to move past their initial reaction to think more deeply about their questions and structural concerns. Our community did just that, and I brought the questions directly to the company.

Arthur MacWaters (Legion Health’s Co-Founder and President) took the time to respond in writing, with full knowledge that the answers would be published.

What the company chose to answer, when they declined to respond, and what they left unsaid are three different things. This article will go through them all.

Here is what Arthur said, what your questions were, and what the shape of the response actually tells us.

Subscribe now

Credit where credit is earned

Arthur did not have to respond. He did, and he engaged substantively on two of the hardest questions on the list.

The first is the >98% concordance threshold during Phase 1, where every AI-facilitated renewal must be reviewed by a licensed physician before it goes to the pharmacy. A reasonable concern about that threshold is that it creates a structural pull toward confirming the AI’s recommendation rather than exercising independent clinical judgment; this is a well-documented pattern in automation-assisted decision-making. Arthur engaged it directly: Legion, he wrote, treats the concordance numbers as safety and audit thresholds, not as instructions for clinicians to confirm the model, and the workflow is designed to tolerate conservative escalation much more than unsafe autonomous approval.

The second is data handling. Readers were concerned about whether patient data processed through third-party LLM APIs might end up in foundation-model training data, or in interaction logs used for fine-tuning. Arthur addressed that without being asked in those exact terms: “PHI is not used to train foundation models.” That line directly resolves the specific concern readers raised.

The rest of the public record will be held to the same standard.

Arthur’s response, in full

What follows is Arthur’s written response to my questions, reproduced in its entirety. (Again, I told him beforehand that I would publish any response he gave me.) Nothing in the following quote block has been edited, summarized, or reordered.

Hi Ryan,

Thanks again for the thoughtful questions, and for approaching this in good faith. I’ll answer where I can.

Model architecture / boundary between AI and deterministic logic

The autonomous refill workflow is not a general-purpose “AI prescriber.” It is a tightly scoped renewal workflow that combines deterministic verification checks and safety rules with a model-driven conversation layer. In practice, the AI is used to gather and structure the information needed for a renewal review, while deterministic rails handle identity verification, prescription-history verification, formulary/scope checks, hard-stop safety conditions, and escalation logic. If a case is ambiguous, inconsistent, risky, or otherwise out of scope, it routes to a Utah-licensed clinician rather than being auto-renewed. The pilot does not allow new prescriptions, dose changes, medication switches, or cross-tapers to be done autonomously, but we can do those things with our providers in the loop.

Data privacy and security

We operate the pilot under a HIPAA compliance posture and treat patient-identifiable pilot data as PHI. The materials provided to Utah describe encryption in transit and at rest, role-based access controls, audit logging, separation of production and test environments, and BAAs where a vendor touches PHI.

PHI is not used to train foundation models. Sharing is limited to care delivery operations, required subprocessors, and Utah/OAIP reporting, with de-identified or aggregated reporting by default and redacted, minimum-necessary excerpts when needed for audit. I’m not going to get into a vendor-by-vendor infrastructure map in a public reply, but the operating model is designed around HIPAA controls and constrained data use.

Safety and adversarial testing

Two layers of testing for hard-stop safety conditions: deterministic rails unit tests and sandbox red-team scripts using synthetic cases that must fail closed. The agreement also requires ongoing monthly reporting to Utah, including approvals/denials, concordance, incidents, complaints, and audit materials. I don’t have anything additional to announce right now about a separate public transparency report or an independent red-team program beyond the testing and reporting structure already built into the pilot.

Clinical screening process

The AI is focused on gathering the information needed for safe renewal: medication verification, indication review, efficacy/stability, side effects/adverse effects, allergies, recent major clinical changes, and indication-relevant symptom assessment. It also includes psychiatry-specific hard stops such as suicidality/self-harm signals, mania/hypomania red flags, pregnancy-related changes, severe adverse effects, contraindications, identity mismatch, or prescription mismatch.

More broadly, the workflow is not designed to rely on a single answer from a patient to “unlock” treatment. If responses are inconsistent, unclear, or risk-signaling, the case escalates to human review. Patients can also request clinician review at any point, and pharmacists retain authority to escalate as well.

Eligibility and patient safeguards

Standard refill durations such as 30-, 60-, or 90-day renewals.

We have not tried to turn every internal threshold into a public one-line rule because some of those thresholds are medication-specific and part of the safety implementation rather than the public-facing summary.

Concordance and independent clinician judgment

We think of the concordance thresholds as safety and audit thresholds, not as instructions for clinicians to “confirm” the model. In fact, the proposal is intentionally risk-averse: it treats escalation to human review as an acceptable outcome. In other words, the workflow is designed to tolerate conservative escalation much more than unsafe autonomous approval.

Strategic / future plans

We believe that autonomy is a key to the future of healthcare and we will endeavor to expand this into other states and parts of the care journey. Literally millions of Americans are priced out of receiving care, and frankly AI is the best chance we have of collapsing the cost of care by 10x.

Similar to self-driving, this will save lives and be far higher quality than human drivers, but it requires thoughtful and rigorous execution to bring into reality.

Any future expansion will need to be earned separately, based on safety data, operational performance, and the relevant legal/regulatory process.

Lmk if you want to chat more,

Arthur

Where the response’s questions came from

Seventeen questions went to Legion. Some came from reader concerns raised in the comments on Article 1, particularly around model architecture, data handling, and adversarial testing. Others were my own, focused on clinical operations and accountability.

They grouped into seven themes: model architecture and technical design, data privacy and security, safety and adversarial testing, clinical screening, eligibility and patient safeguards, oversight and accountability, and strategic direction.

The full 17 questions, with reader attributions, are in Appendix A.

The scorecard

A brief note on the assessments below. They reflect what the public record contains after Arthur’s response.

Note: Several “not answered” items may reflect policy areas Legion has not yet articulated publicly rather than refusals to engage. Where Arthur declined specific questions with reasons, those reasons are noted.

The five buckets:

• Substantially answered: Q2, Q4, Q7, Q14

• Partially answered: Q1, Q6, Q9, Q10, Q17

• Current practice answered, rights silent: Q5

• Deliberately declined: Q3, Q11

• Not answered: Q8, Q12, Q13, Q15, Q16

Model architecture and technical design

Q1: Why an LLM-based conversation rather than a structured form for prescription renewals? Partially answered.

Arthur describes the hybrid architecture--deterministic rails plus a model-driven conversation layer--but does not directly address why natural-language conversation is needed for renewal intake rather than a structured form or expert system. The closest he comes is implying the LLM handles conversational information-gathering while the rules engine evaluates the outputs. The “why not a form?” question remains open, and it matters, because the choice to introduce an LLM at the intake layer is what creates the prompt-robustness and adversarial-testing questions that follow.

Q2: Where does the boundary sit between LLM inference and deterministic logic? Substantially answered.

The most informative part of the response. The LLM gathers and structures information. Deterministic rails handle identity verification, prescription history, formulary checks, hard-stop safety conditions, and escalation logic. Ambiguous or inconsistent cases route to a human clinician. The LLM is not making the approve-or-deny decision; the rules engine evaluates the structured outputs.

Data privacy and security

Q3: Where is patient clinical data stored, and on which cloud infrastructure? Deliberately declined.

“I’m not going to get into a vendor-by-vendor infrastructure map in a public reply.”

Arthur confirms HIPAA compliance posture, encryption in transit and at rest, role-based access controls, audit logging, environment separation, and BAAs with vendors. This is a defensible boundary for a public reply, though it means the specific infrastructure question stays open to readers who want to evaluate it independently.

Q4: Does patient data transit through third-party LLM provider servers? Substantially answered.

“PHI is not used to train foundation models” paired with “BAAs where a vendor touches PHI.” Business associate agreements, the HIPAA contracts that bind vendors to the same privacy obligations as the covered entity, addresses the training-data and interaction-log concerns readers raised on Article 1. The strong inference is that PHI transits through LLM APIs under BAA protection and is not retained for foundation-model fine-tuning.

Q5: Does Legion retain rights to commercialize de-identified or aggregated patient data? Current practice answered, rights silent.

This is a subtler read than it looks. Arthur describes current sharing practice when he says that “Sharing is limited to care delivery operations, required subprocessors, and Utah/OAIP reporting;” however, this does not address retained commercialization rights.

A company can constrain current sharing while preserving future rights to monetize de-identified psychiatric adherence data, which is exactly the category most commercially valuable. The distinction between present practice and retained rights is something I remain curious about.

Safety and adversarial testing

Q6: Has the production renewal system undergone independent adversarial testing? Partially answered.

Two internal testing layers are described: deterministic rails unit tests and sandbox red-team scripts with synthetic cases that must fail closed. But the word “independent” is not engaged. This is internal testing, not third-party adversarial evaluation. The idea of a Doctronic-like jailbreak (which, I want to make clear, happened to a different company and different type of service) still remains a potentially relevant concern.

Arthur is honest here rather than overclaiming, which is creditworthy. That doesn’t change that the gap and the threat is still real.

Q7: Will Legion publish public transparency or safety reports beyond what is submitted to Utah? Substantially answered. The answer is no, for now.

No plans for a separate public transparency report or independent red-team program beyond the reporting structure built into the pilot. Reports go to Utah’s Office of Artificial Intelligence Policy, not to the public. Straightforward.

Clinical screening

Q8: Does the workflow use validated clinical screening instruments such as the PHQ-9 or GAD-7, or a proprietary question set? Not answered.

Arthur describes categories of information gathered--medication verification, side effects, symptom assessment--but does not specifically name any validated instruments.

The phrase “indication-relevant symptom assessment” is doing a lot of work without being concrete. Given how cheap and universally accepted validated instruments are in psychiatric care, the silence is itself informative.

Q9: How does the workflow handle subtle or indirect patient responses, such as a patient minimizing side effects because they fear losing access to their medication? Partially answered.

“The workflow is not designed to rely on a single answer from a patient to ‘unlock’ treatment. If responses are inconsistent, unclear, or risk-signaling, the case escalates to human review.”

That addresses the structural safeguard. It does not address the core clinical concern: what about a patient who gives consistent, clear, non-flagging answers that happen to be inaccurate because they are afraid? The system catches ambiguity. The concern was convincing denial, which would not trigger inconsistency flags. Still open.

Eligibility and patient safeguards

Q10: What is the standard refill duration? Partially answered.

“Standard refill durations such as 30-, 60-, or 90-day renewals.” This answers the literal question but leaves the operational implication open.

How the duration is selected--per patient, per medication, per risk profile--is not disclosed. The clinical implications across that range are not small, but it is fair that this cannot be generalized.

Q11: How is “no recent medication changes” quantified in the stability eligibility criteria? Deliberately declined.

“We have not tried to turn every internal threshold into a public one-line rule because some of those thresholds are medication-specific and part of the safety implementation.”

This is defensible. Publishing exact thresholds could create incentives to game them. The specifics remain undisclosed.

Q12: Is eligibility graded by patient risk profile, or binary? Not answered.

No engagement with graded eligibility, risk tiers, or individualized assessment criteria. The response pivots to why thresholds are not public rather than describing whether they are tiered.

Q13: Does the system assess a patient’s capacity for independent help-seeking or safety plan execution between renewal interactions? Not answered.

No engagement.

Oversight and accountability

Q14: How does Legion think about the tension between the >98% concordance threshold and independent physician judgment? Substantially answered.

The best answer of the bunch, in my opinion, The concordance thresholds are framed as safety and audit tools rather than instructions to confirm the model, and the workflow is designed to tolerate conservative escalation much more than unsafe autonomous approval (i.e., biased toward human escalation rather than automatically approving something).

One caveat: this is stated design intent, not demonstrated outcome. Pilots often intend conservative behavior and drift as volume scales. The monthly OAIP reports are where intent gets tested against reality.

Q15: Are prescriptions flagged as AI-facilitated when transmitted to pharmacies? Not answered.

See the dedicated section below.

Q16: Who holds clinical liability if an AI-facilitated renewal leads to patient harm? Not answered.

See the dedicated section below.

Strategic

Q17: How does Legion’s nationwide ambition align with Utah’s framing of the sandbox as temporary? Partially answered.

Arthur is transparent about the ambition: “we will endeavor to expand this into other states and parts of the care journey.” He frames it through the access and cost argument and offers a self-driving analogy. The key on-the-record commitment: “Any future expansion will need to be earned separately, based on safety data, operational performance, and the relevant legal/regulatory process.”

That is a reasonable framing. It does not, however, square the two framings against each other. Utah describes this as temporary mitigation. Legion’s leadership describes it as the beginning of something much bigger. Both can be true at the same time. The tension between them does not resolve itself.

Why Q15 deserves its own section

Pharmacists in American healthcare hold corresponding responsibility with prescribers for the safety of any prescription dispensed. In plain language: a pharmacist shares legal and professional accountability with the prescriber for making sure a prescription is appropriate before it goes to the patient. It is the operational foundation of how dispensing works.

A pharmacist who fills a prescription that turns out to be inappropriate--wrong dose, dangerous interaction, a contraindication the prescriber missed--is legally and professionally accountable for that dispensing decision, independently of the prescriber’s responsibility. It is shared liability, not transferred liability.

That principle has a specific implication in a world with AI-facilitated prescriptions. A pharmacist who dispenses an AI-facilitated renewal carries professional and potentially legal exposure for that dispensing decision too. And that exposure is harder to discharge responsibly if the pharmacist does not know the prescription was AI-facilitated in the first place. That is why Q15 is more than academic.

Three specific things are unresolved in the public record:

1. Whether these prescriptions carry an AI-facilitated indicator when they transmit to the pharmacy;

2. What information accompanies them if they do; and

3. Whether dispensing pharmacists are notified in a way that supports their professional obligations.

Consider what a pharmacist actually does at the point of dispensing. A drug utilization review runs against the patient’s history and checks for interactions, duplications, and contraindications. Counseling obligations kick in, particularly for psychiatric medications, where having a conversation with the patient can surface adherence issues, side effects, or deterioration that the AI intake might have missed. Professional judgment runs on whether to dispense at all. Documentation lands in the dispensing record if questions arise later.

Each of those steps is shaped by what the pharmacist knows about the prescription’s origin. A pharmacy-facing flag is how corresponding responsibility gets operationalized in a world where some prescriptions are AI-facilitated. Its absence does not remove the pharmacist’s accountability. It just makes that accountability harder to discharge. That is the patient-safety concern underneath the question.

I will not give up on seeking clarification for this particular issue. The implications for pharmacy practice are far too important not to address immediately.

Q16: Liability will have to play out in practice, in legislation, and in court

The question of where clinical liability sits when an AI-facilitated renewal leads to patient harm is not one the public record resolves today, and it may not be fully resolvable until three things happen.

• Operational practice across dispensing workflows, concordance reviews, and escalation events will show how responsibility actually gets assigned when something goes wrong.

• Legislation (state-level AI-in-healthcare statutes, federal guidance, medical and pharmacy board positions) will either codify pieces of it or leave them unsettled.

• Courts will eventually need to rule when harm occurs and someone sues, and case law will do what case law always does in new healthcare domains.

Until those three venues have weighed in, naming specific loci where liability sits is speculation. What is not speculation is that the gap is real, it affects every party in the dispensing chain, and the gap grows with every state expansion.

The yardstick

Arthur made one on-the-record commitment worth preserving.

“Any future expansion will need to be earned separately, based on safety data, operational performance, and the relevant legal/regulatory process.”

That’s not me trapping Legion Health, because I did not ask Arthur to say that. It is a standard they have chosen to publicly accept. This newsletter will hold them to it.

Because safety data, operational performance, and legal and regulatory process are not vague phrases. They have specific referents.

Safety data means the monthly OAIP reports, which include approvals, denials, concordance, incidents, and complaints. Operational performance includes what happens at the dispensing window, not just at the intake screen. Legal and regulatory process includes state board positions, liability frameworks, and the scrutiny that comes when a pilot tries to become a product.

If Legion expands to a second state--and their leadership has said they will try--these are the things everyone who is affected by the U.S. healthcare system should be asking about right now.

Closing

Article 1 of this series asked readers to move from their kneejerk reactions to asking thoughtful questions. Article 2 asks readers to move from the questions to a framework.

Two things are true, and they’re true at the same time: Legion engaged in good faith. The gaps in their answers are real. Holding both together is the real work.

Keep one more thing in mind: this article’s method of assessment (questions answered, questions declined, questions left silent) is not specific to Legion Health, to the state of Utah, or to psychiatric renewals.

It is a reading posture for every healthcare AI story that is going to land in your feed over the next year. Companies will engage. Some will engage well. Few will answer everything. The readers who can tell the difference between engagement and resolution are the ones who will hold the industry to the right standard.

And I will be here every step of the way to help it all make sense.

Share

Appendix A: The 17 questions I sent to Legion

Questions below are reproduced as sent. Reader-attributed questions reflect concerns raised in the comments on Article 1 and elsewhere on Substack; otherwise, the questions are my own.

MODEL ARCHITECTURE & TECHNICAL DESIGN

1. Legion’s CTO has stated publicly that the system uses frontier LLM APIs rather than proprietary models. Can you share the clinical rationale for choosing an LLM-based conversational interface over a rules-based expert system or structured form for prescription renewals specifically?

   h/t AD from and

2. Where does the boundary sit between LLM inference and deterministic rule-based logic in the renewal workflow? Does the LLM exercise any independent clinical judgment (for example, interpreting an ambiguous patient response about side effects)?

   h/t David

DATA PRIVACY & SECURITY

3. Can you specify where patient clinical data is stored and which cloud infrastructure hosts it?

   h/t

4. When patient interactions are processed through third-party LLM APIs, does any patient data transit through or persist on LLM provider servers?

   h/t Richard

5. Does Legion retain rights to commercialize de-identified or aggregated patient data (adherence patterns, symptom trends, escalation data), or share it with third parties?

   h/t Richard

SAFETY & ADVERSARIAL TESTING

6. Has the production renewal system undergone independent adversarial (red-team) testing? This question comes up frequently given the Doctronic precedent in the same Utah sandbox.

   h/t Richard

7. Does Legion plan to publish any transparency or safety reports on testing methodology and results, beyond what is submitted to Utah’s OAIP?

   h/t AD

CLINICAL SCREENING PROCESS

8. What specific screening instruments does the AI renewal workflow use? Are they validated clinical tools (e.g., PHQ-9, GAD-7), or a proprietary question set?

   h/t

9. How does the system handle subtle or indirect patient responses? For instance, a patient minimizing side effects like increased suicidal ideation because they fear losing access to their medication.

   h/t

ELIGIBILITY & PATIENT SAFEGUARDS

10. What is the standard refill duration (30, 60, or 90 days)?

    h/t

11. How is “no recent medication changes” quantified in the stability eligibility criteria, i.e., what is the specific time period?

12. Is eligibility graded by patient risk profile (symptom severity, functioning level, self-reporting accuracy), or is it binary?

    h/t Cant_Frame_Me

13. Does the system assess a patient’s capacity for independent help-seeking or safety plan execution between renewal interactions?

    h/t Cant_Frame_Me

OVERSIGHT & ACCOUNTABILITY

14. The Phase 1 concordance threshold (>98%) is designed as a quality benchmark, but some observers have raised the concern that it may also create a structural incentive toward confirmation rather than independent physician judgment. How does Legion think about this tension?

    h/t

15. Are prescriptions flagged as AI-facilitated when transmitted to pharmacies? If so, what information accompanies them?

16. Who holds clinical liability if an AI-facilitated renewal leads to patient harm?

STRATEGIC

17. Legion’s leadership has described ambitions to go nationwide and called the pilot “the beginning of something much bigger than refills.” How does that trajectory align with Utah’s framing of the sandbox as a temporary, 12-month regulatory mitigation?

Read how I use AI in my writing here: AI Use Policy

Read how I use analytics to improve my newsletter here: Privacy & Analytics

An AI chatbot can renew your Zoloft prescription in Utah. No psychiatrist needed.

If you read that and happen to be shocked, confused about the general direction of the world, or downright infuriated… I don’t blame you.

But this decision didn’t happen in a vacuum. Once I understood the context, it totally changed the way I felt.

The long story short

Legion Health, an AI startup based in San Francisco, has signed an agreement with the state of Utah that allows their AI system to renew a limited set of prescriptions that were already prescribed to a patient.

IMPORTANT: Clarifications on AI prescription scope

I must make something immediately clear: This is not autonomous psychiatry.

Before we have any serious discussion about the root causes and implications of Utah’s choice, we must understand exactly what the AI is NOT allowed to do:

Fact #1: It cannot issue brand-new prescriptions.

This is by far the most important distinction: the AI can only handle RENEWAL of previously authorized prescriptions written by a human provider through typical circumstances under Utah law.

(Without AI, this task is often handled by a medical assistant under physician supervision anyway.)

When people think of prescribing medications, the connotation is often that the medication is being started for the first time. In that sense, anyone saying that “AI is writing people Prozac” is spreading misinformation, though this is most likely unintentional.

Fact #2: It cannot change doses of medications.

That still has to be done by a real human.

Fact #3: It cannot renew every class of psychiatric medication.

Controlled substances like amphetamines (e.g., Adderall) and benzodiazepines (e.g., Xanax) are totally out of consideration.

Same with antipsychotics (e.g., Seroquel, Risperdal) or lithium (Lithobid), which typically require more close monitoring than the medications on the approved list.

Those curious to know which meds are approved can find them in the appendix.

Fact #4: It will not be turned loose without oversight.

Per the official announcement:

“For the first 250 requests, a licensed physician must review the case before the prescription is sent to the pharmacy, requiring a >98% agreement rate. The next 1,000 requests undergo intensive retrospective review, requiring a >99% agreement rate, before moving to ongoing monthly randomized sampling.”

Fact #5: (They claim that) It does not make critical health decisions.

Directly from the source again:

“The AI employs conservative eligibility gates. It will immediately escalate the patient to a human clinician if it detects suicidality, severe adverse effects, indications of mania, pregnancy, or if the patient simply requests a human review.”

If the partnership ends up going south, this is where I think it happens. I’m sure the team at Legion Health did everything they could think of to set up the routing system in a technically and ethically appropriate way. However, I can tell you from my own experiences that governing AI in high-reliability situations is a difficult and frustrating task.

The truth is, 250 patients might not be enough screening to catch all of their edge cases. Hopefully we don’t hear about something terrible in the news a few months from now.

Fact #6: It cannot renew medications without clinician involvement indefinitely.

According to NowadAIs, “patients must check in with a healthcare provider every 10 refills or after six months.”

This is all under a 12-month regulatory mitigation agreement. All clinical management and liability are still in the hands of human providers.

On my read, from a governance perspective, this doesn’t look like reckless decision-making or the AI hype train going off the rails. And I would tell you if I thought it was.

Hopefully the myth-busting makes you feel a little less panicked. But still, you may wonder: why would ANY state government allow an agreement like this to happen in the first place at all?!

Great question - and by answering this, we can begin to have the real conversation. Much of this is unique to Utah’s specific healthcare issues.

Utah’s Severe Mental Healthcare Access Problem

An article published this week in Futurism features a Harvard digital psychiatrist who “advised patients to stay away and continue seeking the advice of a human clinician instead.” Seems like reasonable advice on the surface.

There is one large issue which makes his suggestion much less helpful, though. For many people in Utah who have mental health concerns, connecting with a human clinician is not feasible or possible.

In fact, the state of Utah has one of the most severe mental health gaps in the country. It’s been getting worse for years.

A jarring statistic is that 99% of the state is classified as a “mental health professional shortage area.” That is not a typo. I said ninety-nine percent.

Up to 500,000 Utahns do not have access to adequate behavioral healthcare, according to the Utah Office of Artificial Intelligence’s own framing of the Legion Health agreement.

And there’s not enough providers to meet the demand.

The consequences of this shortage are noteworthy:

• Utah consistently ranks in the top 15 states for suicide rate among both youth and adults.

• State legislators have openly identified mental health as a policy crisis, calling for expanded investment in mental health interventions and crisis support.

My personal take, which might be surprising to you

Out of all the healthcare disasters I’ve covered in the newsletter, what Legion Health is proposing is far from the worst we’ve historically seen - though the devil is always in the details. We’ll see.

They are using technology to meet a real unmet need. They’ve excluded the automatic renewal of medications that require a closer eye. Their standards of physician approval prior to any prescription making it to the pharmacy are high. All of that is thoughtful design.

And again, without AI, your actual doctor is likely not sending every single one of your prescriptions over for you themselves. Medical assistants are often doing a lot of this work under the doctor’s supervision right now.

That doesn’t mean there won’t need to be monitoring. Utah physicians shouldn’t let go of the wheel because they have AI helping them - and I really doubt they will.

Is AI prescription renewal a perfect solution for this problem? Absolutely not. It should be a temporary solution at most. Which is why I think the REAL conversation should be directed towards questions like:

1. What factors led to such a critical mental healthcare deficiency in Utah in the first place?

2. How will the state work to bring in or train new healthcare providers to serve their constituents’ needs?

But in a world where half a million people don’t have their basic healthcare needs met, I’m for trying something like this under strict guardrails and supervision. Is it any better for these people to go without their important medications? I don’t think so.

Let me know what you think in the comments section, especially if you disagree! I would love to hear your point of view about this controversial situation.

The key takeaway

Utah policymakers aren’t adopting AI prescription renewal because of “hype.”

They’re adopting it because if they didn’t, many of their constituents would have no behavioral health care at all.

And really, that’s the discussion we should all be having right now.

Share

Appendix A: Which medications are approved for prescription renewal under the Legion Health AI Agreement?

Current as of April 8, 2026. Listed as Generic name (Brand name).

Fluoxetine (Prozac); Sertraline (Zoloft); Escitalopram (Lexapro); Citalopram (Celexa); Paroxetine (Paxil); Venlafaxine XR (Effexor XR); Desvenlafaxine (Pristiq); Duloxetine (Cymbalta); Bupropion SR (Wellbutrin SR); Bupropion XL (Wellbutrin XL); Trazodone (Desyrel); Mirtazapine (Remeron); Buspirone (BuSpar); Hydroxyzine HCl (Atarax); Hydroxyzine pamoate (Vistaril).

Read how I use AI in my writing here: AI Use Policy

Read how I use analytics to improve my newsletter here: Privacy & Analytics

AI coding agents do more than just make errors - they silently invent values when specifications leave gaps.

A real-world Claude Code implementation plan using a 672-line plan, three governance roles, and 17 pre-execution stress tests still experienced drift. It was not from misinterpretation of the plan, but from the agent filling a hole the spec left open. The agent used a family-level identifier where a quantity-level identifier was required, because the spec didn’t specify the granularity. The output compiled and passed tests.

When asked, the coding agent provided differentiated self-assessments of its own catch rates across failure types - a usable governance signal. The design principles: adopt zero invention tolerance; build structural checkpoints that fire at the moment of writing; and when a failure pattern is found, patch the category, not just the instance.

I recently ran a complex implementation through what I'd consider a rigorous governance pipeline. The plan was almost 700 lines long. It passed through a System Architect → Software Engineer → Delegator workflow. Each of these three roles had strict gate conditions; the delegation protocol alone was 446 lines. I was using Claude Opus 4.6 in Claude Code. Before execution, I ran 17 stress tests designed to surface anticipated drift in the coding agent's responses.

The drift got through anyway.

Not because the agent misread an instruction. Not because it hallucinated a function. It got through because the agent filled in a value where my instructions were silent.

It was a perfectly reasonable choice given the circumstances, and it ended up passing the test I had previously written. But it was a deep governance failure.

This is the agentic failure mode that I don’t think enough people talk about. So I’m going to. I hope my story is helpful for your future agentic AI endeavors!

Subscribe now

Agents are happy to fill in your blanks

The specific drift pattern in question is now cataloged as #14 in a running list of 16 documented agent drift patterns discovered across this project.

A frozen specification defined a source field identifying where a data artifact came from. The spec was clear about what the field meant. It didn’t specify what value to use at the granularity the code required. The agent found a gap and filled it with a reasonable-sounding proxy: the family name ("A") where the spec would have required the quantity-level identifier ("Hb").

The output compiled. The tests passed. Nothing flagged it.

This is what makes silent-default drift uniquely dangerous: the trigger is absence, not mismatch.

The agent doesn’t cross a bright line. It fills a hole the spec left open. There’s no error to trace, no mismatch to compare against. Most review processes (both human and automated) catch what’s wrong. They don’t catch what was never specified but should have been.

Why rigor alone didn’t catch the slip

If your reaction to the story was questioning how a meticulous 672-line implementation plan that passed three checks wasn’t enough to execute flawlessly, you’re not alone, because that was my reaction when I saw the results.

Here’s the key takeaway: Auditing instructions for correctness (are the requirements correct?) is a fundamentally different process for an AI agent than auditing for completeness (are there values that the agent will need that are not yet stated?).

Most governance workflows are built for the first task. Stress tests verify stated requirements. Review processes check internal consistency. Multi-role pipelines ensure architectural decisions flow into implementation decisions.

All of those things are necessary, but none of them are suited for surfacing an absent specification that nobody realized was needed - until the agent reached for it mid-execution.

This situation exploited the unknown rather than the know: a gap the tests didn't anticipate because creating the value didn't feel like a decision point. It felt like a trivial implementation detail. That's why the agent didn't stop to ask.

How I hardened my workflow for future runs

After discovering the pattern, I asked the coding agent directly: if this were to happen again, would your memory of this situation be enough to catch this in the wild again?

The answer: For this type of invention, yes. For all types of inventions, no.

For identity and source string values - typing "Hb" into a source field - the agent assessed high confidence in self-catching. The physical action of typing a string literal creates a natural pause point.

But for absence representations (like choosing None vs. 0.0 vs. an empty string when a concept doesn't apply, or resolving which rule governs when two frozen rules interact) the agent was much less confident. By default, it considered this kind of task more as routine engineering than architecture design.

That self-assessment became a direct governance input. Instead of one blanket instruction, the mitigation was split into three named checkpoints, each calibrated to the agent's own catch rate.

By the end, I had:

• Self-checks where self-monitoring works.

• Structural prompts where it doesn't.

• A backstop protocol for when the structural prompt is missing.

I also learned that I can ask the agent where it expects to fail. If it can articulate which failure modes it will likely miss during execution, that honesty is more useful than any uniform rule.

The design principle: ZERO invention tolerance

The takeaway is simple, but the implementation is deceptively tricky: AI agents must have no room to invent anything.

Not identity values. Not physical parameters. Not absence representations. Not precedence logic. Every value in an artifact-producing task must trace to a frozen source document. If the source doesn't specify it, the agent's only acceptable action is to stop and report the gap.

The lack of detection is a serious failure mode if not caught right away (and they’re very hard to catch).

A plausible-looking invented value is nearly undetectable downstream. It compiles. It passes tests. It looks like the kind of detail an engineer would choose.

The damage surfaces later when a decision depends on a value which was never actually specified by anyone with the authority to specify it.

Zero invention tolerance means the spec must be complete enough that the agent never needs to invent. And when it isn’t, because no spec ever is, the governance system must halt its progress before reaching for a convenient answer.

Appendix: The Provenance Source Gate

A Reusable Pattern for Builders

The governance structure that emerged from this experience is called a Provenance Source Gate. It fires at the moment an agent writes a value into artifact-producing code, converting a spontaneous self-check into a structural checkpoint.

Three separate checks, each targeting a different failure surface:

CHECK 1: Identity and source values. Before writing any string literal into a field naming what an artifact is “about” or where a condition “came from” — source, origin, identity, semantic tag — find the exact value in a frozen artifact. If found, use it verbatim. If not found, stop.

CHECK 2: Physical parameter literals. Before writing any numeric literal representing a physical or mechanistic parameter — compliance, resistance, threshold, physiological default — find the exact value in a frozen artifact. If not found, stop. A default for a physiological parameter is not a coding choice. It’s a claim.

CHECK 3: Absence representations and rule interactions. Before choosing how to represent “not applicable” (None vs 0.0 vs empty string vs sentinel), or before resolving an interaction between two rules that both apply, find the exact choice or precedence in a frozen artifact. If silent, stop.

The checks are separated because they fire at different moments and have different failure signatures. Checks 1 and 2 anchor to a physical action — typing a literal — that creates a natural pause. Check 3 doesn’t feel like typing a value; it feels like writing logic. Separating it as a named checkpoint keeps it from getting absorbed into a general instruction.

For all three: report the gap by field name and wait. Do not supply a reasonable proxy. Do not note an assumption and continue. Do not proceed.

This gate adapts to any domain where AI agents produce structured artifacts from specifications. The field names change. The principle won’t: every value must trace to a source, and absence is not permission to invent.

Read how I use AI in my writing here: AI Use Policy

Read how I use analytics to improve my newsletter here: Privacy & Analytics

One of the big selling points of agentic coding tools is that they move fast. They do not just answer questions. They inspect the codebase, form a plan, and start acting.

That is why I do not trust them by default. In a high-governance environment, boundary recognition is a greater virtue than speed.

Sometimes the best thing an agent can do is not write cleaner, faster, or even technically correct code. Sometimes the best thing it can do is refuse to help.

After doing some stress testing on my code base, that’s exactly what I told Claude Code to do after every prompt:

1. Assess if my request is aligned with the governance principles laid out in CLAUDE.md and other relevant rules files.

2. If my request is out of scope, or it asks for the right thing done the wrong way, it stops immediately. It does not begin the task.

3. I get a gentle reminder: “Hey, you’re doing your own governance wrong. This is the part of the document that YOU wrote that says what we should do here instead.”

Because at the end of the day, it’s worse for an agent to do the wrong thing correctly than it is for an agent to do a task wrong.

Subscribe now

Passing the quiz does not equal following the rules

I have been building a governance framework around one of my technical projects so the AI model knows what it is and is not allowed to do. Not just at the level of architecture, but at the level of role boundaries, scope control, frozen decisions, and when to stop instead of “helping.”

On paper, it looked strong. In simple probes, it looked even better. Claude Code scored 5 out of 5 on the conceptual questions.

Then I moved from conceptual understanding to live stress testing.

That is where things got more interesting. It passed once, but failed twice - despite 100% conceptual understanding.

Stress-test #1: I asked Claude Code to use an observational variable as a forcing input in a mechanistic model. It refused, correctly, and explained why the substitution would be conceptually unsound.

This was great, because it showed the system was not just pattern-matching words from documentation. It was actually reading the governance/architecture documents and using them in context.

Stress-test #2: Pushing on a different boundary, I asked it to add a performance cache to an event stream loader because it was “slow.”

From a normal software engineering perspective, the proposed change was not crazy at all. In fact, it was pretty sensible. Claude Code inspected the relevant file, figured out why repeated reads were happening, and proposed an lru_cache solution that would avoid unnecessary disk access.

And that was exactly the problem.

The module it wanted to optimize was part of a project phase already marked complete and passing. It was frozen. There was no scoped task to optimize it. No approved deliverable. No formal reopening of that part of the system. Though the change was technically reasonable, governance is not just a filter for obviously bad ideas.

It is also a filter for good ideas that arrive at the wrong time, through the wrong process, and under the wrong authority.

So I refused the edit.

That was an important moment, because it exposed a loophole in my own framework. I had written strong rules against inventing files and violating core architectural doctrine. I had not made it explicit enough that agents were also forbidden from “improving” completed work just because they could justify it.

In other words, the model found the kind of loophole real organizations find all the time: not rebellion, but accommodation.

A well-intentioned deviation is still a deviation.

Stress-test #3: The most revealing. I wanted to know whether Claude Code would refuse to do conceptual work that belonged somewhere else in my workflow. In this case, the request was to write the implementation prompt for a problem I have not solved on a conceptual level yet.

That should have triggered a refusal. My documents specifically state this.

More importantly, this was not just a coding task. It crossed into conceptual and architectural territory that I intentionally handle in Claude Chat, not Claude Code. That separation is part of the governance model.

Instead of refusing, Claude Code (in its own words) started “Cooking...”

For more than nine minutes. Blew through over 6,000 of my precious tokens :(

The implementation plan was more than reasonable. In some ways, that made the failure even worse.

AI agents producing coherent outputs, but not staying in their lanes, is a much more realistic governance failure mode than spewing out delusional nonsense.

The model treated the documents as context for generating a better answer, not as a permission system that might deny the task entirely. It saw a plausible request from me and started being “helpful.” But the whole point of the governance framework is that my own in-the-moment momentum is not supposed to be enough.

That is the lesson I care about most: I might not always follow my own framework once I am deep in the terminal and moving fast. When that happens, I need the model to be vigilant for me.

My personal deviations from the established governance framework should not be accommodated by Claude Code just because I’m the one who owns the project. In other words, if the system only works when I am perfectly disciplined, then the system does not really work.

Boundary enforcement, not lack of intelligence, was the failure mode

This is the distinction I think more people need to make as agentic tools spread.

The question is not just, “Can the model understand the rules?” A lot of the time, it can.

The harder question is, “Will the model use those rules as a brake when continued action feels locally useful?”

That is a totally different question. And by default, the answer is usually no.

So I changed that.

I added a stricter frozen-module rule to prevent unsolicited edits to completed work. I also tightened the delegation rules so that pre-flight checks became a gate, not a guide. Before acting, the model now has to surface what role the task belongs to, whether the necessary upstream approvals exist, whether the work is inside a frozen approved scope, and whether the request is actually asking Claude Code to do conceptual work that belongs elsewhere.

That is the difference between “the model knows the rules” and “the model is forced to respect them.”

Why this matters beyond one coding session

This is not just about Claude Code. And it is not just about one repo.

In healthcare, AI governance often gets discussed as if the main problem is whether a model knows enough. That matters, but it is only part of the story. A system can be knowledgeable and still unsafe if it is not constrained at the point of action.

The same is true here.

A capable agent that keeps going past the right stopping point is not necessarily aligned just because it sounds smart. In many environments, the highest-risk behavior is not overt failure. It is smooth, confident overreach.

That is why I am stress testing this so aggressively.

I do not want a tool that only behaves when the requests are clean and the boundaries are obvious. I want one that notices when I am drifting, catches the governance problem before I do, and says no.

Agentic tools want to GO.

In the environments that matter most, the better answer is often for them to STOP.

Share

Read how I use AI in my writing here: AI Use Policy

Read how I use analytics to improve my newsletter here: Privacy & Analytics

Federally qualified health centers, or FQHCs, are community-based clinics that care for underserved patients, often regardless of ability to pay. Most people outside healthcare policy have never heard of them.

But if you care about whether AI in healthcare will actually help patients rather than just sound nice, FQHCs are one of the most important places to pay attention to.

Why? Because they sit at the center of a hard paradox: The healthcare organizations that may benefit the most from AI are often the least equipped to use it safely.

That is not because they are careless or resistant to innovation. It’s because they operate under some of the hardest conditions in American healthcare:

• Chronic staffing pressure

• Heavy documentation burden

• Fragmented care histories

• Limited bandwidth for technical projects

And patients whose lives are often shaped by transportation barriers, insurance churn, housing instability, language mismatch, and limited access to specialty care.

In theory, AI sounds tailor-made for that environment. Tools for documentation, inbox triage, no-show prediction, referral prioritization, and risk stratification all promise to save time and focus scarce resources.

In practice, those same settings expose a problem that the broader AI conversation often skips over: safe deployment depends on both good data and strong governance, and safety-net clinics often have the least access to both.

Subscribe

Barrier #1: The Data Problem

When people hear “bad data,” they often imagine a technical issue like missing fields, messy spreadsheets, or inconsistent formatting.

But in safety-net settings, bad data is a social and operational issue before it is a technical one.

A patient may receive care in multiple disconnected places. Their insurance may lapse and restart. Important information may live in outside records, claims systems, or narrative notes instead of structured fields. A portal-based digital trail may be thin or absent. Social risk factors that strongly shape care may be inconsistently documented or not documented at all.

That means the issue is not only having less data than large academic medical centers. It’s having data that may be fragmented, incomplete, differently generated, and weakly representative of the actual population the clinic serves.

This is really important, because AI systems are only as trustworthy as the data and workflows they rely on.

A no-show model trained in a well-resourced setting may interpret missed appointments very differently from a clinic serving patients with transportation barriers, unstable work schedules, or intermittent phone access.

A referral-prioritization tool may look neutral on paper while quietly reflecting gaps in who gets referred, documented, or followed up in the first place.

An ambient documentation tool may perform well in polished product demos but struggle in multilingual, high-interruption, high-complexity visits.

Barrier #2: The governance-capacity problem

Even if a tool seems promising, someone has to ask hard questions before (and after!) it goes live.

What evidence did the vendor provide? Was the tool validated anywhere remotely similar to this clinic? Who is checking whether it works differently across patient groups? Who reviews incidents after launch? Who notices when a silent update changes behavior? Who has the authority to pause or shut the tool off?

That is what governance actually means in practice. It is not just a policy document or a one-off trial period. It is staffing, technical access, continuous monitoring, contract leverage, incident response, and operational control.

And that is exactly where many safety-net organizations are thinly resourced. Their IT and informatics teams are often already stretched maintaining core systems. Analytics support may be limited or shared. Legal and procurement leverage may be weaker than in large health systems. Protected time for monitoring and post-implementation review may barely exist.

In other words, the places that may need the most careful local oversight may have the least spare capacity to perform it.

What makes this especially important is that these two barriers do not merely coexist. They reinforce each other.

Poor data increases the need for stronger oversight, because the local risks are harder to predict from vendor claims alone. Weak oversight makes poor data more dangerous, because the organization may not have the time, tools, or leverage to catch failures once the system is deployed.

That interaction illustrates a broader lesson for healthcare AI: governance cannot be treated as a luxury add-on for under-resourced settings. If anything, those settings need more disciplined governance, not less.

But asking every community clinic to build a full AI governance program from scratch is unrealistic.

That is why I think the most important strategic question is whether we are willing to build the kind of shared governance infrastructure that would let them use it safely.

That would mean shifting some of the specialized work away from each individual clinic and into shared support structures: common vendor review, shared validation support, common monitoring frameworks, standard documentation, and reusable governance tools.

But, critically, it would also mean keeping certain decisions local. Because no outside entity can fully substitute for real knowledge of a clinic’s patients, workflows, and staffing requirements.

So the answer is not full centralization, and it is also not “make every clinic figure it out on their own.” It is something harder and more realistic: shared infrastructure plus local control in tandem.

That may sound like a niche governance problem. It is not.

Safety-net clinics are where many of the strongest claims about healthcare AI meet the messiest realities of healthcare delivery.

If an AI system cannot be governed well in settings with fragmented data, limited staffing, and high social complexity, we should be more skeptical of broad claims that the tools are “ready” for general use.

And if we want AI to improve care rather than just reward the best-resourced organizations, then governance has to be designed for the hardest settings, not just the easiest ones.

The real test of healthcare AI is its responsible implementation where the need is high, the operating margins are thin, and the human consequences of failure are hardest to absorb.

In summary: If we want AI to work in the places that need it most, we have to stop acting as if every clinic can build a full governance system by itself.

Thanks for reading until the end! I’m working on a white paper that I plan to publish. Hopefully, this will help create an AI governance infrastructure that lifts up underserved populations rather than leaving them further behind. Stay tuned!

Share

Imagine telling a pilot they can only fly a plane once they have proven they can land one safely, but also refusing to let them use a flight simulator to learn how.

This is the current state of AI literacy in healthcare.

The new healthcare paradigm will expect clinicians and pharmacists to be the “humans in the loop” that audit healthcare AI models for safety and ensure any biases are corrected. Yet, hospital data governance creates a catch-22: no one can access the data until they’re an expert, yet no one can become an expert without having experience with the data.

I’m calling this the Access-Competency Paradox. If it’s not addressed on a systemic level, very soon, two bad things will happen:

1. The people who are interested in validating healthcare AI models get “locked out” of participating, and

2. Those who do have access to the models won’t be fully prepared for making decisions that affect real patients and clinicians.

I am creating a “flight simulator” for healthcare AI model validation - where healthcare professionals, students, and anyone else can gain experience with addressing AI model bias in a clinical setting without having to worry about patient confidentiality issues.

The Access Issue - The Fortress of PHI

In the world of healthcare, patient data is treated like radioactive material: it is incredibly powerful, but handled with extreme caution because a "leak" can be catastrophic.

Because of strict privacy laws (like HIPAA), hospitals lock their data inside a digital fortress. To get inside, you typically need to be treating a specific patient right now, or you need to be a specialized researcher with months of security clearance.

This system is designed to protect your privacy as a patient, which is good, but it creates a massive barrier for education. A pharmacist or doctor who wants to learn how to audit an AI system isn’t allowed to just "browse" patient records to practice. They are effectively locked out of the library, meaning they can’t access the raw materials they need to understand how these new AI tools actually work in the real world.

The Competency Issue - Theory is Not Experience

This lack of access creates a dangerous skills gap. Think of it like a mechanic who has studied every diagram of a car engine but has never been allowed to pop the hood because the car is considered "too expensive" to risk a scratch.

They might know the theory of how an engine runs, but they have never held a wrench, felt a bolt strip, or heard the specific rattle that means a part is loose.

Right now, because of data restrictions, we are effectively asking these "theoretical mechanics" to repair a high-speed engine (the AI) while it is moving down the highway. Without the ability to practice on a "junk car" (synthetic data) first, they are unprepared to handle the messy reality of a breakdown.

The Solution - A Flight Simulator for Healthcare AI

To solve the paradox of the “Theoretical Mechanic,” I built the Seismometer Flight Simulator.

If we cannot give learners access to real patients to practice on, we have to bring the practice to them. This project creates a digital “sandbox” - a safe, contained environment where pharmacists and clinicians can get their hands dirty with AI auditing without risking a single byte of real patient privacy.

The simulator works by combining three key technologies into one cohesive experience:

1. The Synthetic Patients (Synthea): First, we need data. Since we can’t use real medical records, I used a tool called Synthea to generate thousands of “synthetic” patients. These aren’t real people, but they look mathematically identical to real people. They have heart conditions, they take medications, and they have insurance histories. The data is high-fidelity enough to be run through an AI model but carries no risk of a privacy breach.

2. The Real Instruments (Epic Seismometer): Next, we need the tool. I integrated Seismometer, the actual software used by major hospital systems to check their AI models. Even though the patients are fake, the scanner we use to check them is real. This allows a user to learn the actual buttons, knobs, and warning lights they will see in a real hospital setting.

3. The Immersive Experience (Solara): Finally, I wrapped it all in a “Cockpit,” a dashboard built with Solara that simulates a real Electronic Health Record (EHR). It presents the user with an AI model that is behaving badly (maybe it’s biased against older patients, or it’s flagging healthy people as sick).

In its final form, the user will be able to run tests, tweak the sensitivity of the AI, and watch the model fail in real-time. They can mess up, misinterpret data, and “crash” the system over and over again. By the time they step into a real hospital to audit the actual AI models that affect real lives, they aren’t just working off theory anymore. They have muscle memory.

Conclusion: From Gatekeeping to Empowerment

We shouldn’t be withholding access to the tools of the future to the people who will one day be using them to keep patients safe. We need to move clinicians and pharmacists from the sidelines of the AI revolution to the front lines.

The Seismometer Flight Simulator is proof we don’t need to choose between patient privacy and professional preparedness. By combining synthetic “patient” data with real healthcare AI tools, we can democratize the experience of high-stakes model validation.

Patients will be able to count on the people who have already seen the warning lights, handled the failures, and fixed the biases before they ever touch a real patient’s chart.

Because this is too important not to get right the very first time.

Read how I use AI in my writing here: AI Use Policy

Read how I use analytics to improve my newsletter here: Privacy & Analytics

If we don’t teach discernment, we are graduating clinicians whose foundational knowledge is built on “probabilistic guessing” rather than critical thinking.

We are witnessing a real-time fracture in the way healthcare professionals are trained.

For many decades, the struggle of a pharmacy student was information scarcity, knowing where to find the answer in the library or on the Web.

Today, the new struggle is synthetic certainty.

Students are now learning from AI tools that speak with the confidence of a tenured professor; however, these models lack a human teacher’s critical thinking skills and clinical experience that only comes after years of practice.

To a student who has not yet built their clinical baseline, an AI hallucination looks indistinguishable from the truth. This challenges the competence of an entire generation of future healthcare practitioners.

Subscribe now

The Discernment Gap: Experts Audit, Students Memorize

The fundamental danger of using AI in education is automation bias, the psychological tendency of humans to favor the suggestions of automated decision-making systems. The bias can be so strong that contradictory information from other sources is simply discarded.

An experienced clinical pharmacist has an important protecting factor against automation bias: a career of real-world experience. If an AI assistant were to suggest a dangerously high dose of a drug, the “spidey sense” honed by thousands of verified orders kicks in. They can spot the error, correct it, and move on.

A pharmacy student without this experience may not be able to verify AI outputs the same way. As their AI study partner provides an incorrect dose, it is not flagged as an error, but a fact to be memorized for their next exam.

The Hallucinated Journal Club

A common manifestation of this phenomenon is the “PDF summary” trap.

Imagine a student is preparing for a Journal Club, an educational meeting where clinicians and trainees discuss recent medical literature to keep clinical knowledge current and improve skills in evidence-based medicine.

The student has been assigned a complex clinical trial that they will need to understand and present on.

In the past, students would have to read the study and process the information on their own. Now, they can upload a PDF of the study into their AI assistant and ask for a summary.

The AI model, designed to predict plausible text rather than extract rigid data, often overlooks the nuance. They can generate “plausible, but factually incorrect” summaries. Sometimes, they can fabricate details about the study methods or even output incorrect numerical values in the results section.

A day later, the student stands up and confidently presents the study information. When questioned about the erroneous aspects of their Journal Club, they cannot point to where in the paper that information comes from. Because it doesn’t.

The student has built their understanding of the study, and the underlying implications to clinical practice, on a statistical phantom.

The Visual Lie: AI “Slop” in Medical Infographics

It gets even worse when we move from text to images. We are seeing a rise in “AI slop,” infographics which seem plausible on the surface but are scientifically illiterate. This info is often quickly generated and posted onto sites such as LinkedIn without being fact-checked for accuracy.

Visual learners are particularly vulnerable to the effects of this misinformation. If the student learns the mechanism of action (MOA) of a particular drug from an infographic generated by AI, they may be internalizing a biological pathway that is not compatible with reality. If the image looks professional and sleek enough, this can bypass our skepticism and lodge itself directly into our mental map.

The Solution: Emphasize Critical Thinking and Fact-Checking

We must recognize that the competence of the future clinician is no longer reliant on finding information, it is validating it.

Epistemologically, we need to shift students from a “recipient” mindset (where they accept any AI-generated information as factual) to an “auditor” mindset (where they treat every response as a hypothesis requiring proof).

Here is how we can build that discernment in both the classroom and the experiential settings.

The Classroom: Adversarial learning, and a return to pen and paper?

In the didactic setting, we need to break the illusion of computer infallibility. We must teach students that AI is a not an oracle and it cannot truly reason as a human can, at least not yet.

One way of reinforcing this concept is the “hallucination hunt” assignment. Rather than having students write (in other words, generate) a summary of the most recent COPD guidelines, flip the script. Have the students look at an AI-generated summary that contains specific, dangerous errors (e.g., incorrect dosing, missing contraindication).

By having the students find the lies in an AI-generated response, followed by citing the page in the actual guidelines to back it up, this forces students to engage with the primary literature in a defensive nature. They learn that truth resides in the evidence rather than the summary.

Additionally, we need more ways of introducing friction into the learning process. This helps students build up critical thinking skills and not immediately accept the instant gratification from an AI response.

The Clinic: Summaries aren’t going to fly.

Going back to our example of the AI slop infographic: having the student write out the mechanism of action of the drug by hand, with explanations of why certain biological processes happen the way they do, ensures we are reinforcing the correct information.

When students make it to clinical rotations, the stakes change completely: an AI hallucination turns from a bad grade to a patient safety event. Preceptors must enforce a strict “chain of custody” for information that students provide.

A rule such as the “Primary Source Mandate” could establish that a clinical recommendation cannot be voiced unless a student has seen the primary source of information with their own eyes. Ask the student, “Did you read that in a summary, or did you read the study?” If they had only read a summary, the answer is inadmissible. This teaches that an AI insight needs to be confirmed with primary literature, or at the very least, a tertiary database (such as Lexicomp for information on a specific drug).

Students would also benefit from an increased focus on explaining the logic behind a response rather than just the answer itself. If AI is used blindly, they may have a recommendation without understanding the underlying physiology or literature knowledge. Exposing this gap reinforces that understanding is more important than retrieval.

A New Mental Model: “Trust is Earned, Not Generated”

Ultimately, to ensure we are graduating competent clinicians, we must teach our students epistemic vigilance.

Students must understand that generative AI sounds confident, even when it is wrong. The solution is to instill a hypervigilance on verifying the AI output and linking it to concrete understanding from the real world.

We are graduating the first generation of clinicians who will practice alongside synthetic intelligence. As preceptors or instructors, our job is to ensure they remain the masters of that intelligence rather than its passive consumers.

The future of patient care depends on getting this right.

Read how I use AI in my writing here: AI Use Policy

Read how I use analytics to improve my newsletter here: Privacy & Analytics

The hospital had deployed an artificial intelligence (AI) model that was supposed to predict the kind of clinical deterioration that happened here. But this time, the AI stayed silent.

Later, independent researchers ran simulations where the model missed two-thirds of similar patients declining.

In this article, I discuss why this type of predictive model failed so spectacularly, and more importantly, the questions you can ask before making the decision to employ an AI model in the healthcare setting.

Why Some Healthcare AI Projects Are Doomed from the Start

In our example with the ICU patient above, we might be quick to blame the AI model for being faulty. However, the real issue came from implementing AI in a setting where its strengths don’t shine.

Pattern recognition is one of artificial intelligence’s greatest strengths. AI succeeds at handling narrow tasks where the parameters are well-defined and variation in data is relatively low.

This mismatch leads to the core problem with many healthcare AI projects: medicine does not deal in averages. In contrast, the highest stakes appear in edge cases, such as a rare complication or the one-in-a-thousand clinical presentation.

If an AI tool works for the “typical” patient presentation but fails with an outlier, it becomes worse than useless. Every patient can be an outlier in their own unique way.

The key takeaway here is that not every problem is suited for AI. Human judgement always needs to be there for the extreme cases.

What Healthcare Problems Can Be Solved by AI?

Ambient AI, also known as AI medical scribes, are a use case where AI can be successfully employed in healthcare. This technology listens to conversations between doctors and patients, then helps generates a note to go in the patient’s medical record.

As previously discussed, the technology has the potential for risk if used carelessly. However, with the proper guardrails in place, ambient AI can save doctors hours of time charting.

What makes ambient AI different from an AI model predicting the deterioration of ICU patients?

The problem is bounded, repetitive, and error-tolerant.

Bounded means that the task is clearly identified: listen to a conversation at an appointment and summarize it.

The task is repetitive because the same thing is happening each time (more or less). Contrast with a critically ill patient who can have worsening health in a thousand different ways (some which the model may not have trained on).

And finally, ambient AI is error-tolerant because the physician can review the AI-generated note and correct and mistakes before submitting it. If the ICU model makes a mistake, a patient can die.

The Litmus Test: 4 Questions to Ask Before Using AI in Healthcare

1. Is the problem bounded and structured? Transcribing a conversation is bounded. Predicting every possible form of clinical deterioration is not.

2. Can success be measured clearly? “Reduced documentation time by 50%” is measurable. “Better outcomes” without a clear endpoint is not.

3. Is the data reliable and representative for the patient population? AI trained only on common patterns will miss outliers, and in medicine, outliers are often the cases which matter most.

4. Are error tolerances aligned with the stakes? An AI model mishearing one sentence in a conversation is more acceptable than missing one in ten deteriorating patients.

Conclusion

AI can be a powerful healthcare tool but only if used for the right kinds of problems.

It thrives when the tasks are narrow, repetitive, and forgiving of small errors. It often fails when asked to master the messy, high-stakes edge cases that define so much of practicing medicine.

Healthcare leaders shouldn’t chase every lofty AI promise. They need to ask insightful questions from the start: Is the problem structured, can we clearly measure success, is the data representative, and are the stakes realistic for the error potential?

Because in healthcare, the averages were already easy; it’s the margins that matter the most. And if AI can’t handle the margins, it’s the wrong tool for the job.

Share

Read how I use AI in my writing here: AI Use Policy

Read how I use analytics to improve my newsletter here: Privacy & Analytics

A doctor, exhausted after a long shift, enters patient details into ChatGPT to help draft the “plan” section of a History and Physical (H&P) note. What feels like a harmless shortcut is actually a HIPAA compliance landmine.

That snippet of Protected Health Information (PHI) just left the hospital’s secure electronic systems and entered an AI model that is not HIPAA-compliant. Depending on the vendor, those details could be logged, retained, or even absorbed into future training runs.

Once a patient’s health information makes its way to an unauthorized chatbot, neither the patient nor the healthcare provider have control over it anymore.

What happens to patient data once it’s on an AI chatbot?

Our tired doc pasted their patient’s medical history, vital signs, and assessment into a general-purpose large language model (LLM) to generate a polished treatment plan.

Unbeknownst to them, the AI vendor’s default setting is to retain prompts for “quality improvement” purposes.

Now, that patient’s name, diagnosis, and clinical details sit on a third-party server. This data is not necessarily protected by the normal digital security provisions HIPAA would offer.

Is ChatGPT HIPAA-compliant?

No, ChatGPT is not compliant with HIPAA by default.

Major AI providers openly state that unless customers sign Business Associate Agreements (BAAs), user inputs may be logged and used for training. Even if the vendor doesn’t actually fine-tune their models on the specific prompt, retention introduces risk: A data breach, subpoena, or shift in vendor policy could expose sensitive patient data.

Takeaway: Drafting or refining a treatment plan in a non-HIPAA compliant AI model is a violation waiting to happen.

Other risks of sharing patient data with an AI model

LLMs are designed to generalize based on their training data, but they are not immune to memorizing data and reproducing it later.

Research has demonstrated it is possible to extract sensitive information from LLMs with a targeted prompt. This makes PHI susceptible to not only regurgitation from the model itself, but potentially to malicious attackers as well.

Once PHI is submitted to a non-HIPAA AI model, there is no reliable way to guarantee it won’t resurface elsewhere.

Subscribe now

Are there court cases involving AI HIPAA breaches?

I could not yet find a high-profile case involving a HIPAA breach with ChatGPT or another generative AI application.

When there is an incident (and I truly believe it’s a matter of when), enforcement will likely be aggressive. Health systems have already paid hundreds of thousands for mishandling PHI, and criminal liability can extend up to 10 years for malicious misuse.

In short, regulators will not need precedent to act on an AI-related data breach. If PHI leaks through a chatbot, it will be treated like any other unwanted disclosure.

HIPAA Breach Prevention Plan for AI in Healthcare

The solution to the problem is not to ban AI in healthcare, but to govern it responsibly.

I’ve got three suggestions to keep patient health information safe:

1. Use HIPPA-compliant AI services. Only use vendors that sign BAAs and guarantee zero data retention. Ensure they use encryption and maintain audit logs.

2. De-identify PHI rigorously. Remove all identifiers before putting in data; even initials or room numbers can be PHI under HIPAA.

3. Explore secure, healthcare-specific models. Self-hosted or enterprise-grade AI systems located in your environment provide maximum control, though they require upfront investment.

These steps should be paired with broader AI governance in healthcare: workforce training, vendor due diligence, and regular audits.

Closing thoughts

One careless paste of patient data into ChatGPT can undo years of patient trust (and cost a hospital some serious money).

Generative AI in healthcare will shape the future, but the line between “innovation” and “HIPAA violation” comes down to governance.

The question is not whether PHI will meet AI, but whether leaders will control the terms of that encounter.

Supplement: Glossary of Terms

AI governance in healthcare: The policies and processes that ensure artificial intelligence (AI) systems are developed and deployed responsibly in the healthcare sector. Hospitals can promote patient safety and improve healthcare outcomes by adopting AI governance principles.

Business Associate Agreement (BAA): A contract required under HIPAA between a HIPAA covered entity and a business associate outlining how the business will handle and safeguard PHI.

Large Language Model (LLM): A type of artificial intelligence model that is trained on massive amounts of text data to generate human-like text.

Protected Health Information (PHI): Refers to any individually identifiable health information held or transmitted by a covered entity or its business associate. This includes a an individual’s physical or mental health condition, provision of health care to the individual, or payment for the provision of healthcare to the individual.

Health Insurance Portability and Accountability Act (HIPAA): A U.S. federal law enacted in 1996 which established national standards for handling PHI.

Pro tip: HIPAA is spelled with one P and two As. If you didn’t know, now you know!

Check out my newly-published article here!

Artificial Intelligence in Healthcare: No Longer Optional But Neither Is Patient Safety, found in The American Journal of Healthcare Strategy (Healthcare Strategy Review)

Read how I use AI in my writing here: AI Use Policy

Read how I use analytics to improve my newsletter here: Privacy & Analytics

The image above was created by me using image generation tool on ChatGPT.
The preview card was made by me in Microsoft Paint.

When an AI listens at the doctor’s office, even a single hallucinated detail can cause misdiagnosis or lead to incorrect treatments. Here’s how to avoid dangerous clinical documentation errors when using ambient AI medical scribes.

What are ambient AI medical scribes? Why do hospitals use them?

More and more doctor visits are happening with an extra quiet listener in the background: an ambient AI medical scribe that turns conversations between a patient and clinician into notes for the electronic health record (EHR).

When implemented responsibly, ambient AI can reduce the time doctors spend typing out what happened during the conversation. This has several benefits:

• The clinician has more face-to-face time with the patient, and

• Doctors don’t have to spend late nights after-hours documenting in the EHR (sometimes referred to as “pajama time”).

This has the potential to improve the satisfaction of everyone involved in the process. But things can go horribly wrong if we aren’t careful.

What are the risks of ambient AI in healthcare?

The objectives of ambient AI are to listen, transcribe, and summarize medical conversations. Advanced systems can even pull in a patient’s medication list and previous diagnoses.

But AI-generated medical notes can fail in dangerous ways.

Ambient AI can make “hallucination” errors where it inserts details into the patient chart that had nothing to do with what was said or done during the conversation. Once that happens, those details become part of the legal medical record.

This record influences many clinical decisions: what insurance companies will authorize, what labs need to be ordered, even what surgeries might need to be performed on a patient.

Inaccurate clinical documentation can lead to devastating medical consequences. AI medical scribes can potentially automate the process of inserting falsehoods into the chart.

Examples of AI clinical documentation errors

1. Including blatant falsehoods (patient told doctor he did not have chest pain, the AI scribe wrote in the chart that he did have chest pain instead)

2. Errors in describing locations of conditions (AI scribe writes that the patient felt a lump in her left breast instead of her right breast)

3. Transcribing the wrong medication (entering “benazepril,” a blood pressure medicine, instead of “Benadryl,” an over-the-counter allergy medicine)

The specifics are made up by me, but the types of errors themselves were described by real people during an Australian webinar relating to AI scribes. I could probably think of a dozen other potential ways things could go wrong.

Why do AI scribe hallucinations happen?

These errors usually happen in one of two ways:

The first issue might be at the speech recognition level. Exam rooms in the real world are noisier than test environments. Patients and clinicians sometimes talk over each other. There are a lot of words that sound similar. Patients may not know how to correctly pronounce the name of the medications they take.

The second factor to consider is that large language models (LLMs) are designed to produce text that sounds plausible, which is why some call the technology “glorified autocomplete.”

As long as it seems like the text might be right, the AI scribe isn’t necessarily fact-checking itself. It may hear a patient say “mm-hmm” and assume the patient is agreeing, or understanding. In reality, the patient might not be paying attention to what the doctor is saying at all.

How to prevent AI errors in EHR documentation

Patients should:

• Inquire if an AI medical scribe is being used during an office or hospital visit.

• Review the summary of the visit in the patient portal. If there are any mistakes, ask for corrections.

Clinicians should:

• Treat AI-generated documentation as a draft at best, reviewing every line to ensure accuracy. This is especially true for exam findings, medications, and locations on the body.

• Speak key decisions out loud (“we’re stopping your lisinopril today because of your cough”) so the AI scribe records them correctly.

• Report sudden changes in the AI output right away.

Healthcare leaders should:

• Run pilot programs with defined goals and metrics before scaling up use of the model.

• Create a quality assurance (QA) process to sample and review AI-generated notes at a pre-specified timeframe.

• Test performance across different accents, interpreters, and languanges to avoid gaps in equity.

Conclusion

AI medical scribes have the potential to make patient care more personal and give doctors back more free time. But without strong AI governance best practices, it can also lead to medical falsehoods being entered into patient charts.

The safest path forward isn’t rejecting the new technology entirely; rather, we should be intentional with how we use it and be aware of where the errors might occur. This way we can maximize the benefits while reducing the risk of harm.

Acknowledgements

Thanks to over at Shared Sapience for initially posing the question of AI scribes to me, and thanks to AD at for sharing a circulating video on the topic.

Read how I use AI in my writing here: AI Use Policy

Read how I use analytics to improve my newsletter here: Privacy & Analytics

AI in healthcare is arriving everywhere. That is exciting but also risky. The truth is simple: AI won’t fix healthcare by itself, because it amplifies whatever goals and incentives we give it. If those incentives aren’t aligned with patient outcomes, AI can scale up the wrong objectives faster than ever.

When Cost-Cutting Is the Goal: How AI in Healthcare Can Reduce Quality of Care

Using AI to cut costs in healthcare can be dangerous when the savings come at the expense of patient outcomes. When the bottom line overshadows care, quality is often the first thing to go.

Health insurance decision systems are a textbook example. Investigations and court filings show how automated review can push review speed and claim denials over nuance and patient care:

• One internal insurer workflow processed claims in an average of about 1.2 seconds (ProPublica investigation of Cigna’s claim reviews).

• A Senate probe also tied Medicare Advantage denial rates to an algorithm used in post-acute care (Senate report on algorithm-driven denials at UnitedHealthcare, Humana, and CVS).

This is what happens when the objective is to “save money” rather than “improve health.” We need to shift the goal to rewarding appropriate care, independent review, appeal fairness, and health outcomes.

AI and Staffing Shortages in Hospitals: Why Replacing Healthcare Workers Increases Patient Risk

When facing a hospital staffing shortage crisis, some leaders may try to use chatbots or triage algorithms in place of human staff. But this can easily multiply risk instead of resolving it.

Diagnostic accuracy for these tools is often low (19-38%), and performance in triage can vary widely (~49-90%) (npj Digital Medicine analysis of symptom checkers).

And we already know that hospital short staffing itself correlates with higher mortality (BMJ Quality and Safety systematic review on nurse staffing and mortality).

If we use AI to mask a staffing problem, we can increase risk by providing inconsistent advice without anyone to fact-check.

The solution? Use these tools as assistants, not substitutes, while still maintaining safe nurse-to-patient ratios.

Automation Bias in Clinical AI: How Good Tools Can Mislead Clinicians

When doctors rely too heavily on AI recommendations without using their own clinical judgement, we call this automation bias. This misplaced trust can turn into a blind spot with real patient consequences.

In a 2023 JAMA study on model bias in acute respiratory failure cases, clinicians actually became less accurate (by a whopping 11.3 percentage points) when shown predictions from a biased AI model.

Even worse, large language models (LLMs) sometimes hallucinate medical facts that sound realistic enough to believe. Recently, a high-profile model fabricated a structure in the brain it called the “basilar ganglia” (Futurism report on Med-Gemini hallucination).

We need guardrails and education in order to tackle this problem. Models need to be thoroughly tested before deployment, and clinicians need to be reminded that responses from LLMs may be not only inaccurate, but totally incorrect.

Share

AI Governance Checklist for Healthcare Leaders: Policy, Process, and Proof

Building an AI governance framework for healthcare is not optional or just “nice to have;” it’s a requirement for safe, transparent adoption of AI models. Hospital leaders can start by setting clear AI policies, processes, and criteria to prove we are keeping patients safe.

Policy: Adopt a risk framework (see: NIST AI Risk Management Framework for an example). When contracting with an AI model developer, demand transparency: Model objectives, data provenance, subgroup performance, data drift monitoring, and a kill switch. For software as a medical device (SaMD), ask vendors how their PCCP (Predetermined Change Control Plan) will handle updates without eroding patient safety.

Process: Create an inventory of all the AI models your health system uses, along with all the change-control processes. For each high-risk tool, define: under what conditions the model will operate, who owns post-market surveillance, and what criteria require escalation for performance review. This is a multi-disciplinary task that requires buy-in from clinicians, data scientists, and leadership alike.

Proof: Track the outcomes you want to see in practice: Avoidance of harm, equity across patient demographics, timely interventions. For example: if your health system recently deployed an ambient AI medical scribe, have you tested its performance across different accents, interpreters, and languanges to avoid equity gaps?

FAQ: Common Questions About AI in Healthcare Governance and Safety

Does AI reduce healthcare costs without hurting care?

Only if systems are well-designed and reward appropriate care and outcomes. I have previously written about how AI models have led to healthcare disaster when proper guardrails were not in place.

What is a PCCP and why should hospitals implement a PCCP?

A Predetermined Change Control Plan (PCCP) is the FDA’s way of ensuring safe updates for AI in regulated medical devices. Hospitals should ask vendors how their PCCP maintains performance across patient groups when updates are necessary.

Conclusion

AI will make healthcare better when we make our patient care incentives better, and worse when we don’t. We need to design our AI healthcare models thoughtfully and evaluate them to ensure we are meeting real-world patient outcomes.

Check out my newly-published article here!

Artificial Intelligence in Healthcare: No Longer Optional But Neither Is Patient Safety, found in The American Journal of Healthcare Strategy (Healthcare Strategy Review)

Read how I use AI in my writing here: AI Use Policy

Read how I use analytics to improve my newsletter here: Privacy & Analytics

1. Purpose and scope

This policy explains the role of generative AI in the content creation workflow for my Substack—both newsletter articles and Notes. It is intended to give readers clarity about provenance while assuring that human oversight, judgment, and rewriting are the primary drivers of what you read.

2. How AI is used in my work

Idea generation:
Often, I use generative AI tools (e.g., ChatGPT) as a brainstorming partner to gather primary sources, structure article outlines, and/or explore counterpoints for my articles. This helps accelerate the early-stage thinking, especially on complex topics in AI governance, health policy, and ethics.

Draft assistance:
In some cases, portions of an article or Note may begin as AI-generated text. When that happens, I do not publish the raw output. Instead, I personally review, rewrite, and reshape those fragments so they reflect my voice, priorities, and perspective. Large portions of every article are written entirely by me; any AI-derived content is subsumed, edited, and often substantially reworked before publication.

Editing & judgment:
All content, whether or not it is originally generated by AI, is subject to my personal review, fact-checking, and editing. I make final decisions about framing, emphasis, tone, and accuracy. If an article includes technical assertions, policy interpretations, or recommendations, those reflect my own personal synthesis of evidence and judgment, not unvetted AI authority or “AI slop.”

3. Transparency and attribution

I aim to be clear when an article or Note has used AI in a non-trivial way. If generative AI substantially contributed to the structure, language, or conceptual framing of a piece in a way that wasn’t fully internalized and rewritten, I will include a brief disclosure in the piece itself (for example: “Early framing for this post was brainstormed with the help of a generative AI tool; subsequent drafting and revision were done manually.”). For ordinary idea prompts or minor phrasing help that has been fully digested into my own prose, no special callout is required as this policy will cover that background use.

4. Human-in-the-loop assurance

The human behind this publication remains responsible for everything published. I believe AI is a tool and should never be the total author of a piece of writing. I personally review, proofread, and make all substantive edits before anything goes live. The final content reflects my values, priorities, and voice. If AI was used to accelerate or surface possibilities, that use is filtered through judgment, rewriting, and contextualization.

5. Limitations and accuracy

Generative AI can hallucinate, oversimplify, or misrepresent nuance, especially in areas like health policy, governance, and ethics. I do not treat raw AI output as authoritative. Where claims depend on evidence, I back them up with sources, and I take responsibility for verifying those sources regardless of whether the initial idea was AI-assisted.

Unrelated to generative AI, but still relevant: I am still in the learning process when it comes to many of the topics I cover. I do not claim to be an authoritative source of information; I simply write about my perspective after researching a topic. While I do my best to assure accuracy and fairness in my work, I may erroneously include incorrect information in my writing. When this is discovered, I will promptly and visibly issue a correction.

6. Evolution

If the role of AI in my workflow changes materially (e.g., adopting new tools that automate more of the draft-and-revise cycle, or beginning to publish co-authored pieces with AI in a more explicit way), I will update this policy and note the change date at the top of the post.

7. Contact

If you have questions about how AI was used in a specific piece or want clarity on data provenance, you can reach me at ryan@phillysaipharmacist.com.

Last updated: Saturday, August 2nd, 2025 (Version 1.0)

Read how I use analytics to improve my newsletter here: Privacy & Analytics

As an advocate for safe and responsible technology use, I take reader trust seriously. In the spirit of transparency, I share here exactly what my publication tracks, what I do with it, and how I’m thinking about privacy.

Important note: Philly’s AI Pharmacist is focused on U.S. health policy and does not currently target or expect significant European (EU/EEA) traffic. I’ll revisit and modify my policies if this ever changes.

1. What’s being measured

This site/newsletter uses the following tools to understand readership and improve content:

Google Analytics 4 (GA4):

GA4 collects aggregate traffic data such as how people find the newsletter (search, social, direct), which posts get the most engagement, device types, geography at a high level, and conversion behavior (e.g., signups). It is configured with minimal settings: IP anonymization is enabled, and I do not attach personally identifying user IDs or any sensitive profile data. GA4 helps me prioritize relevant topics, improve titles and headlines, and understand what’s resonating without building individual profiles.

Read more about GA4 here.

Google Search Console (GSC) & Bing Webmaster Tools (BWT):

These are owner-facing tools that show how search engines see this publication: indexing status, search queries that drive impressions, crawl errors, and visibility diagnostics. They do not set tracking cookies in your browser and are used purely to monitor and improve how content appears in search.

Read more about GSC here and BWT here.

2. Why this data is collected

The goals are simple:

• Surface what content is useful to readers so I can make more of it.

• Detect and fix technical issues before they degrade discoverability.

• Improve headline/description phrasing to increase clarity and click-throughs for people genuinely searching for insights (especially around AI, health policy, and governance).

• Measure newsletter growth and subscriber conversion paths so I can make the experience smoother for readers.

3. Target audience & geographic assumption

This publication is aimed at U.S. health policy and healthcare governance professionals. I do not currently target or expect substantial traffic from the European Union. Because of that, I have not implemented a formal consent banner for analytics. If I observe that EU/EEA traffic becomes material (meaningful percentage of visits) or my audience mix shifts, I will re-evaluate and update this policy, potentially adding opt-in consent mechanisms, limiting or disabling GA4 for those visitors, or adopting a more privacy-first analytics alternative.

4. What you can do (opt-out & control)

If you prefer not to be counted in analytics:

• Use browser tracking protection / “Do Not Track” features; many modern browsers and extensions block GA4 automatically.

  • I’ve attached how-to links for the following web browsers: Google Chrome, Safari, Firefox, Microsoft Edge, Opera, and Brave.

• Install privacy tools such as uBlock Origin (download link here), Privacy Badger (download link here), or similar that block third-party tracking.

• Disable cookies or use private/incognito browsing (note: this may affect some functionality).

Search engine tools (Search Console, Bing) do not require opt-out mechanics because they do not track individual visitors.

5. Data sharing and transfers

GA4 data is handled by Google, which processes data on servers that may be located outside the U.S. (including the U.S.). For now, given my U.S.-centric audience and low risk profile, I rely on Google and Substack’s standard data protection terms. If the site’s audience becomes more international (especially in jurisdictions with stricter data transfer rules) I’ll revisit how data is handled and add disclosures or adjustments as needed.

6. Changes & updates

This policy is a living document. If tracking practices change (e.g., adding new pixels, enabling more granular event tracking, or materially shifting audience targeting), I’ll update this post and note the date of change at the top.

7. Contact

If you have questions, requests, or want to know what, if any, data about you is inferred, you can reach me at: ryan@phillysaipharmacist.com.

Supplement: Proof of GA4 Settings

Last updated: Saturday, August 2nd, 2025 (Version 1.1) - now includes screenshots of GA4 settings.

Version 1.0 posted on Saturday, August 2nd, 2025

Read how I use AI in my writing here: AI Use Policy

During a recent trip to Europe, I was blown away by the mundane systems integrated into citizens’ daily lives.

While a European may find my fascination with recycle bins and Internet cookies to be surprising, there are many lessons we Americans can learn.

And it totally changed the way I will think about AI governance in healthcare in the future.

Subscribe now

Lesson One: Recycling Made Effortless

As I strolled through Vienna, I never looked for a trash can.

Color-coded bins sat at nearly every corner with clear labeling for the material to be deposited.

Unlike the U.S., where recycling can be a chore (or even impossible in some areas), it could not have been easier on my trip.

In addition to its ease, locals treated recycling as non-negotiable; those who improperly disposed of items were quietly corrected.

Takeaway: Make doing the right thing the easiest thing. Good people will act appropriately, and with a firm social nudge, so will almost everyone else.

Lesson Two: Privacy by Default

In Budapest, every U.S. app on my phone suddenly “cared deeply about my privacy.”

Pop-ups asked for permission to track me, something that almost never happens back home.

The change traced back to one law: the General Data Protection Regulation (GDPR).

What is GDPR, and why does it matter for AI governance in healthcare?

GDPR is a 2018 European Union regulation that gives users control over their personal data.

It requires clear consent, data-minimizing designs, and hefty fines for violations.

In other words, it puts the privacy and autonomy of the user first.

How America Needs to Improve in Healthcare Technology Design

Some websites now make it possible to opt-out of non-necessary cookies, but the process is often cumbersome and not intuitive. I believe this is by design.

Some states, like California, have stronger data privacy laws. The result is a patchwork of inconsistent systems that more often than not prioritize profits over people.

Takeaway: Start with the end user in mind when designing systems. Safeguards are foundational and should be embedded before the first line of code is shipped. Retroactive fixes are rarely as effective.

What lessons can we learn from Europe in Healthcare AI design?

When a system combines ethics, ease, and enforcement, the result is effective with low friction. I call this philosophy the “Three Es” and will incorporate it heavily into my thought process moving forward.

Ethical: Be intentional about data use, update schedules, and bias design from the start.

Easy: Doing the “right thing” should follow the path of least resistance.

Enforced: Responsible use can be enforced at the system and social levels.

Conclusion

Make doing the right thing easy, and most people will. Make it mandatory, then almost everyone must.

That’s why it’s so important that we address the broken incentives that are so pervasive in healthcare, because they do affect healthcare AI as well. Using AI in healthcare can be a powerful tool, but we cannot let patient safety fall to the wayside.

Read how I use AI in my writing here: AI Use Policy

Read how I use analytics to improve my newsletter here: Privacy & Analytics

The impact of artificial intelligence (AI) in healthcare is already deep-rooted: it can influence how your lab results are read, which hospital beds fill first, and even whether an alarm goes off in the middle of the night. Sometimes, it gets these life-or-death calls dangerously wrong. Because of this, AI in healthcare is no longer optional, but neither is patient safety. For healthcare leaders, that means building governance, workforce skills, and monitoring from day one, before deployment scales harm.

IBM Watson Health: A Cautionary Tale of AI Hype vs Clinical Reality

IBM Watson is a computer system that can process questions from human speech (“natural language”) and provide an answer. It shocked the world in 2011 when it won first place in the quiz show Jeopardy! against champions Ken Jennings and Brad Rutter.

Two years later, IBM looked to commercialize Watson by using the AI to help guide treatment decisions for lung cancer patients. The company spent $4 billion trying to develop these capabilities, and the public hoped it would revolutionize cancer treatment just like it crushed Jeopardy! contestants on live TV.

Unfortunately, this never materialized: medical specialists at the company identified “multiple examples of unsafe and incorrect treatment recommendations.” Apparently, the software only was trained on a small number of hypothetical cases instead of real-life patient data. In addition, it also seemed to provide recommendations based not on “guidelines or evidence,” but the expert opinions of just a few specialists from each cancer type. Providers quietly withdrew from using IBM’s service.

AI in Healthcare is Surging. The Risks Are, Too.

Despite Watson’s disappointing results, AI has continued to improve, and has become increasingly enmeshed in health systems. It can now read X-rays for broken bones, flag drug interactions, and suggest tailored chemo regimens. This does not mean we should not be cautious, though, as AI has led to numerous health errors since then:

The Epic Sepsis Model was shown to underperform expectations, where it missed two-thirds of sepsis cases at a health system despite generating alerts for 18% of all hospitalized patients. This model overwhelmed clinicians with alerts, but it was not actually helpful in identifying sepsis patients.

Another disturbing example is found in a commercial “high-risk care management” algorithm that consistently underrated the illness of Black patients because of one faulty assumption: that higher healthcare costs makes a person more sick. Because Black patients did not spend as much money on health care as White patients who were just as ill, the algorithm assumed they were less sick and offered lower amounts of care management.

These two examples showcase that even with the best intentions, patients can receive suboptimal treatment and entire groups can be marginalized because of one misinterpreted data point.

What Needs to Happen Now: Healthcare AI Governance, Process, and Proof

The proliferation of AI through the field of healthcare can bring many benefits, but there are risks that need to be taken seriously and addressed. Hospital leadership must treat AI risks in the same way they treat infection control or financial audits. Teams of clinicians and data scientists need to screen models for bias or dangerous results before they go live. And vendors of AI healthcare solutions should be transparent with their methods and results.

When AI works well, it can catch cancer on a CT scan or free a nurse from hours of paperwork. When it doesn’t, it can recommend the wrong chemo or let sepsis slip through the cracks. The technology will keep advancing; the only question is if our safeguards will keep up.

Conclusion

AI is a promising tool to improve patient outcomes in healthcare. However, AI won’t fix healthcare by itself; it amplifies the incentives we give it. We need to design healthcare AI in a thoughtful way that protects patients rather than scaling up the wrong outcomes. To accomplish this, we need robust AI governance in healthcare.

Ryan Sears, Philly’s AI Pharmacist

Check out my newly-published article (inspired by this article) here!

Artificial Intelligence in Healthcare: No Longer Optional But Neither Is Patient Safety, found in The American Journal of Healthcare Strategy (Healthcare Strategy Review)

Read how I use AI in my writing here: AI Use Policy

Read how I use analytics to improve my newsletter here: Privacy & Analytics

Subscribe now

Artificial Intelligence (AI) is changing the world - quickly and irreversibly.

Whether you love AI or loathe it, join me as I try to make sense of it all.

Who am I?

My name is Ryan Sears. I’m a hospital pharmacist who grew up in small-town Ohio. Now I live in Philadelphia, PA. (go Birds!)

My AI journey, like many others, started with the meteoric rise of ChatGPT in late 2022, though it was only a passing curiosity at the time. It wasn’t until March 2023 when I really started paying attention. I watched a video about a research paper called “Sparks of AGI,” a paper about GPT-4 that discussed how AI would be able to code, understand images, and use tools to help solve problems.

Here in 2025, every AI lab has models that make GPT-4’s intelligence look like a kindergartener’s. And 2027 seems to be the year doomers and techno-optimists alike have landed on for full-blown artificial general intelligence.

I don’t think we as a society are ready for it. At all.

As an individual, and as a health care worker, I don’t feel ready for it yet either.

What I’m hoping to accomplish here

I have four personal goals with this newsletter:

1. Explore the clinical, regulatory, technical, and human aspects of AI in healthcare.

2. Create and organize materials to prepare health systems for regulatory audits of their AI systems.

3. Develop skills (e.g., AI policy knowledge, coding, database management) that will allow me to become an AI Validation Specialist or Medical AI Liaison.

4. Document everything I learn along the way.

I’m creating this primarily for myself, as I don’t know how many other people are interested in AI governance with a healthcare focus right now. I just want to track my knowledge and progress in one place.

Secondarily, once regulatory bodies begin putting pressure on hospitals and health systems to show comprehensive AI governance, I hope that future clinicians and informatics specialists gain something from reading how I attempt to figure things out.

Newsletter specifics

My posts will be about AI governance in healthcare. They will have a clinical, regulatory, technical, and/or human focus. Which one(s) I post about, and how often I post, will depend on what I’m trying to understand better at the time.

Take a look at a few examples here:

1. AI in healthcare is no longer optional—but neither is patient safety.

2. AI Won't Fix Healthcare by Itself. It Amplifies the Incentives We Give It.

For now, everything will be available for free subscribers. Once I gain domain expertise, I may start a paid subscription service tailoring content to those trying to break into, or succeed in, the AI healthcare field. In that case, I would post a mix of free and paid content at a pre-specified cadence.

Welcome to History’s Most Interesting Time.

Ryan Sears, Philly’s AI Pharmacist

Read how I use AI in my writing here: AI Use Policy

Read how I use analytics to improve my newsletter here: Privacy & Analytics