Well stated and thought through (like in your earlier piece). As someone also working in this area (for my day job), I do think this will and must change. There is yet to be concerted attention to a crisis to force the bioethical discussions we need to have around where responsibility and accountability lie with AI being used as augmentative and, increasingly, substitution for medical professionals’ judgment. Look at the throughput and I don’t think we can ignore the latter or the push to get to substitution more broadly.
That crisis, accident, etc. will happen. Then the court cases, then the frameworks that are out there will start to be applied and the communication challenges will shift. I do hope those ethical questions can become the norm before all of that, but we are not in that environment with the stressors on the healthcare system as a whole and the promises, inflated or actual, at hand.
That’s a great question, Ryan. I’m going to work a little backwards, because I am on the bioethics and policy side.
I can’t tell you the specific event. What I can tell you is the likely shape it will need to take to actually move things.
It will be a case where the causal chain is clear and understandable to the public. A single patient, a clear harm, and a decision point where a physician either was forced to defer to the AI or was structurally prevented from overriding it. That’s where the accountability gap is undeniable on the face of it.
“The model drifted over eighteen months and outcomes in subgroup X degraded by 6%.” is the more likely, current, and widespread harm, but it will not move policy, because the public cannot see it and the system has too many ways to attribute it elsewhere.
The event that forces the bioethics conversation is the one where responsibility cannot be diffused. Where the physician says they trusted the tool, the vendor says the tool is decision support, the health system says it followed the AI-enabled standard of care, and a patient is harmed anyway. That is the configuration that makes the accountability vacuum visible to people who don’t work in this space, and it is the configuration that produces the court cases and the frameworks getting applied that the original comment was pointing to. Those have mostly been redirected to provider responsibility so far because they still hold all the accountability - but that is not now what these tools market and promote themselves as.
So the failure modes I worry about most are the aggregate ones. The failure mode that changes how we think about it is the acute one with a clear causal chain and no one to hold responsible other than the AI system and rule-infrastructure deferring decisional accountability to it.
Ryan. Thanks for this write up. Clarifying question - I don't see a position from you on whether patients should be able to opt out. Medicine can be so personal and AI pilots do tend to raise skepticism from patients. How does patient opt out fit?
Well stated and thought through (like in your earlier piece). As someone also working in this area (for my day job), I do think this will and must change. There is yet to be concerted attention to a crisis to force the bioethical discussions we need to have around where responsibility and accountability lie with AI being used as augmentative and, increasingly, substitution for medical professionals’ judgment. Look at the throughput and I don’t think we can ignore the latter or the push to get to substitution more broadly.
That crisis, accident, etc. will happen. Then the court cases, then the frameworks that are out there will start to be applied and the communication challenges will shift. I do hope those ethical questions can become the norm before all of that, but we are not in that environment with the stressors on the healthcare system as a whole and the promises, inflated or actual, at hand.
Great thinking and writing!
Thank you very much for your thoughtful comment, Joe!
Completely agree with your assessment that an AI-related patient safety event is a “when” rather than an “if.”
As you’ve suggested, the frameworks I’m trying to build right now are more than likely post-mortem resources serving two purposes:
1. Giving everyday people an explanation as to the conditions leading to the event; and
2. Refuting any future claims that no one could have predicted this happening.
What are the failure modes that you think are most likely without extensive guardrails in place?
That’s a great question, Ryan. I’m going to work a little backwards, because I am on the bioethics and policy side.
I can’t tell you the specific event. What I can tell you is the likely shape it will need to take to actually move things.
It will be a case where the causal chain is clear and understandable to the public. A single patient, a clear harm, and a decision point where a physician either was forced to defer to the AI or was structurally prevented from overriding it. That’s where the accountability gap is undeniable on the face of it.
“The model drifted over eighteen months and outcomes in subgroup X degraded by 6%.” is the more likely, current, and widespread harm, but it will not move policy, because the public cannot see it and the system has too many ways to attribute it elsewhere.
The event that forces the bioethics conversation is the one where responsibility cannot be diffused. Where the physician says they trusted the tool, the vendor says the tool is decision support, the health system says it followed the AI-enabled standard of care, and a patient is harmed anyway. That is the configuration that makes the accountability vacuum visible to people who don’t work in this space, and it is the configuration that produces the court cases and the frameworks getting applied that the original comment was pointing to. Those have mostly been redirected to provider responsibility so far because they still hold all the accountability - but that is not now what these tools market and promote themselves as.
So the failure modes I worry about most are the aggregate ones. The failure mode that changes how we think about it is the acute one with a clear causal chain and no one to hold responsible other than the AI system and rule-infrastructure deferring decisional accountability to it.
Ryan. Thanks for this write up. Clarifying question - I don't see a position from you on whether patients should be able to opt out. Medicine can be so personal and AI pilots do tend to raise skepticism from patients. How does patient opt out fit?