One of the most basic safeguards -- necessary, not sufficient -- for AI and even general technology in healthcare should be transparency. Keep the model a black box (if you really must), but every single input, output, and decision should be logged and available to the patient on demand, no questions asked.

That makes it a lot easier to deal with things like the bad AI transcriptions going around. Get burned by AI that can't hear what you're saying? It's in your file, what it heard and then what it screwed up. Providers don't like everything being on the record like that? Maybe they shouldn't be trusting that tool.

Want to know what the useless AI saw when it sent you to group therapy for acute mania? It's in your file. Providers don't like everything being on the record like that? Maybe they shouldn't be trusting that tool.

LLM can't have legitimacy in doing things to human beings, because it's impossible for LLM to have political legitimacy. Political legitimacy in modern societies is already a vicious zero-sum game with a tiny elite having any in the first place, none will be given to objects. Mental health is inextricably linked with violence and political legitimacy therefore, so the triage lacking efficacy or not doesn't even matter - it doesn't have legitimation. The violence inherent in triage is not justified.

  This change has coincided with a sharp increase in the number of patients who are upset by the time they speak to her. On a typical day, as many as a third of her almost two dozen triage calls are with patients who have struggled to access appropriate care.

It's an amazing and disingenuous posit that the problem here lies with the Algorithm as opposed to the consequences under which it was introduced - cost-cutting. The barriers to care for patients who are already struggling with serious mental conditions are not a new phenomenon related to automated tooling derived from best practice corpus. Rather, they are a direct result of a decline in triage staffing.

Take cardiologist Dr. Lee Goldman’s chest pain decision-making algorithm, originally devised at Cook County Hospital to diagnose heart attacks.

Effectively a poster in the ER illustrating a simple decision tree, it outperformed traditional doctor diagnoses; achieving over 95% accuracy and proving 70% better at identifying patients not having a heart attack. This landmark clinical protocol represented a quantum-leap in outcomes, eventually refined to the Revised Cardiac Risk Index (RCRI) used as the gold-standard today.

So not only are algorithmic approaches old-hat in clinical settings, they are in fact a core component of contemporary best practice medicine. But that's somewhat besides the point when much of the issues raised in the article above refer to issues of executive dysfunction - areas in which low-context Agentic AI are basically designed to remedy, e.g.

// Instead of being referred directly by primary care doctors and contacted to schedule appointments, patients were routed through an app, nonclinical call center staff or simply given a phone number to call. “Now the onus is on the patients to be their own care coordinators,” she said... This dynamic can be particularly challenging for some patients. “Lack of motivation and lack of follow-through are the most common symptoms of depression on the planet,”

So the answer here is clearly an increase in appropriate AI resources, rather than throwing the baby out with the bathwater. If we trust a decision-tree poster to do cardiovascular triage better than the instincts of cardiovascular specialists, and OCR models to interpret results better than veteran Radiologists, why not Agentic AI to do the basic work of scheduling appointments?

> This dynamic can be particularly challenging for some patients. “Lack of motivation and lack of follow-through are the most common symptoms of depression on the planet,” Marcucci-Morris said. “We wouldn’t tell a paraplegic, ‘Hey, walk down the hall in order to get your wheelchair.’”

This was something I really struggled with getting an ADHD diagnosis as adult. I needed an incredible amount of executive function (dozens of phone calls, navigating the mire of health insurance to figure out who I was allowed to go to, who would take me, being bounced around) to be allowed medication to treat my executive dysfunction. I recall wondering how someone less functional than I was was ever expected to get help.

One problem with Kaiser (probably not only Kaiser) when it comes to specialists like mental health workers is that at least in California there was a policy of not allowing specialists to refuse new patients. So over time, specialists' backlogs get bigger and bigger and they are juggling more and more patients and it becomes harder for the specialist to keep track of what's going on. Meanwhile the wait for a new appointment becomes longer as a result - typically months. As a Kaiser patient I personally had multiple specialists quit Kaiser to work elsewhere and at least one cited this as the reason when they told me they were leaving.

It seems really nasty to pair this with reductions in triage staff and the use of algorithmic triage. Their people are already overworked.

What the everliving..... There is no world in which current LLMs are even non harmful for people with actual mental issues.

People making such decisions should be barred from any kind of social adjacent decision-making.