Learning how Utah got here does not require you to agree with their decision. It *will* force you to examine the desperate state of its mental healthcare capacity.
Love the article Ryan, it gave me both food for thought and pause for concern. I appreciate you tagging me in this and want to address it directly and with the same consideration you took writing it.
My focus is civic technology, but there are real intersections here that I can speak to, especially around data handling and model architecture.
Legion's CTO has said publicly that they don't train their own models. It's an API-first system, they seem to prompt third-party LLMs and layer clinical data on top. So the training data is whatever those foundation models were trained on... For a system with prescriptive authority over psychiatric medication, this feels wrong to me. This should be running on closed infrastructure with models specifically trained on clinical datasets, not prompting consumer-facing APIs.
I can't easily access public information about how aggregated psychiatric adherence data, symptom patterns, or escalation trends are handled, shared, or commercialized. No disclosure on where clinical data is stored, whether interaction logs are used for fine-tuning, or what secondary use rights exist for de-identified data? And the previous company approved through this same Utah sandbox was trivially jailbroken & there's no clear indicators Legion has undergone independent adversarial testing either.
The areas I'm building in for civic tech (non-custodial architecture, on-premises data handling, air-gapped AI) could be adapted for HIPAA-compliant clinical use.
I look forward to reading more of your content and learning about how AI is being integrated into these sensitive systems. Thanks again for the engagement!
Thanks so much for weighing in, Richard. I find your insights incredibly valuable - truly responsible AI governance is a cross-disciplinary effort.
I will need to do more digging into the CTO’s comments and the actual architecture of the system. What you’ve mentioned is all very concerning to me.
As many have pointed out, it’s unclear why they went the generative AI route over a rules-based/expert system structure.
Because ultimately, as you point out, no amount of system prompting can override a model’s built-in weights.
I plan on reaching out to Legion Health for clarification on many of these things. If I hear back from them, I will let you know what they say.
Thank you again for your comment. You are right that there are intersections from many domains that play into the issue. I appreciate you sharing your domain expertise!
The only concern I can flag is “patients must check in with a healthcare provider every 10 refills or after six months.” - how long is each refill good for? 30 days? 90 days? 120 Days?
I personally would be risk-adverse to more than a 30 days authorization. But, I am not a doctor, and don't play one on TV.
For stable patients, the two typical quantities for refills are 30- and 90-day supplies. Insurance doesn’t usually cover meds for longer than that.
In the behavioral health setting, fills may even be a shorter supply than that- 7 to 14 days to make sure the patient is tolerating it okay. But initiation/dose changes are beyond the scope of this AI regardless. Just some context for you.
My understanding for the ten refills / six months rule is that it would be whichever comes first, though I haven’t confirmed that with the official docs.
Thanks for reading and for your thoughtful comment! Hope this helps.
“Great news! I am not authorized to do that. 🎉 But I have forwarded your request to a human clinician, a pharmacist, a state regulator, and three layers of retrospective audit. Someone should be with you in 3-7 business months.”
Chatbots can’t ask a question about side effects, notice a subtle change in their patient’s demeanor as they respond and then ask a follow up question. A nurse practitioner or doctor might. This one is personal because psych drugs can feel like a person’s best-last lifeline, and a patient may be reluctant to admit the medicine has, say, increased their occurrence of suicidal ideation, because they don’t want to have the perceived lifeline taken away. Chatbots have demonstrated time and again how they do not perceive subtleties.
Human practitioners can and often do.
The common theme from all of the AI companies is that the training data and algorithms to these models? They are blackboxed. They say they do not know, so they cannot tweak or change them. This makes any medical application untenable in my mind. A pharmacist has their degree hanging on a wall, as do nurse practitioners and doctors in most states. The AI bot has a blank piece of paper.
Love the article Ryan, it gave me both food for thought and pause for concern. I appreciate you tagging me in this and want to address it directly and with the same consideration you took writing it.
My focus is civic technology, but there are real intersections here that I can speak to, especially around data handling and model architecture.
Legion's CTO has said publicly that they don't train their own models. It's an API-first system, they seem to prompt third-party LLMs and layer clinical data on top. So the training data is whatever those foundation models were trained on... For a system with prescriptive authority over psychiatric medication, this feels wrong to me. This should be running on closed infrastructure with models specifically trained on clinical datasets, not prompting consumer-facing APIs.
I can't easily access public information about how aggregated psychiatric adherence data, symptom patterns, or escalation trends are handled, shared, or commercialized. No disclosure on where clinical data is stored, whether interaction logs are used for fine-tuning, or what secondary use rights exist for de-identified data? And the previous company approved through this same Utah sandbox was trivially jailbroken & there's no clear indicators Legion has undergone independent adversarial testing either.
The areas I'm building in for civic tech (non-custodial architecture, on-premises data handling, air-gapped AI) could be adapted for HIPAA-compliant clinical use.
I look forward to reading more of your content and learning about how AI is being integrated into these sensitive systems. Thanks again for the engagement!
Thanks so much for weighing in, Richard. I find your insights incredibly valuable - truly responsible AI governance is a cross-disciplinary effort.
I will need to do more digging into the CTO’s comments and the actual architecture of the system. What you’ve mentioned is all very concerning to me.
As many have pointed out, it’s unclear why they went the generative AI route over a rules-based/expert system structure.
Because ultimately, as you point out, no amount of system prompting can override a model’s built-in weights.
I plan on reaching out to Legion Health for clarification on many of these things. If I hear back from them, I will let you know what they say.
Thank you again for your comment. You are right that there are intersections from many domains that play into the issue. I appreciate you sharing your domain expertise!
The only concern I can flag is “patients must check in with a healthcare provider every 10 refills or after six months.” - how long is each refill good for? 30 days? 90 days? 120 Days?
I personally would be risk-adverse to more than a 30 days authorization. But, I am not a doctor, and don't play one on TV.
Thanks for the note.
For stable patients, the two typical quantities for refills are 30- and 90-day supplies. Insurance doesn’t usually cover meds for longer than that.
In the behavioral health setting, fills may even be a shorter supply than that- 7 to 14 days to make sure the patient is tolerating it okay. But initiation/dose changes are beyond the scope of this AI regardless. Just some context for you.
My understanding for the ten refills / six months rule is that it would be whichever comes first, though I haven’t confirmed that with the official docs.
Thanks for reading and for your thoughtful comment! Hope this helps.
I was on Effexor for a number of years while married to my ex-wife. So I have a passing familiarity with the drugs/concept.
Little chatbot, give me some adderal and something to help me sleep. Thank you!
“Great news! I am not authorized to do that. 🎉 But I have forwarded your request to a human clinician, a pharmacist, a state regulator, and three layers of retrospective audit. Someone should be with you in 3-7 business months.”
🤣
Chatbots can’t ask a question about side effects, notice a subtle change in their patient’s demeanor as they respond and then ask a follow up question. A nurse practitioner or doctor might. This one is personal because psych drugs can feel like a person’s best-last lifeline, and a patient may be reluctant to admit the medicine has, say, increased their occurrence of suicidal ideation, because they don’t want to have the perceived lifeline taken away. Chatbots have demonstrated time and again how they do not perceive subtleties.
Human practitioners can and often do.
The common theme from all of the AI companies is that the training data and algorithms to these models? They are blackboxed. They say they do not know, so they cannot tweak or change them. This makes any medical application untenable in my mind. A pharmacist has their degree hanging on a wall, as do nurse practitioners and doctors in most states. The AI bot has a blank piece of paper.