It's honestly such a big problem. One of my colleagues uses an AI scribe. I can't rely on any of his chart notes because the AI sometimes hallucinates (I've already informed him). It also tends to write a ridiculous amount of detail that are totally unnecessary and leave out important details such that I still need to comb through patient charts for (med rec, consults, etc). In the end it ends up creating more work for me. And if my colleague ever gets a college complaint I have no clue how he's gonna navigate any AI generated errors. I'm all for AI and it's great for things like copywriting, brainstorming and code generation. But from what I'm seeing, it's creating a lot more headache in the clinical setting.

If you're why doesn't this guy just check the AI scribe notes? Well, probably because with the amount of detail it writes, he'd be better off writing a quick soap note.

Is there nothing like HIPAA there or what?

The wilful ignorance and total apathy is appalling.

I've had similar experiences in Australia. I emailed one of my docs' practices asking if they use Heidi AI (or anything similar) and that I do not consent. They were using it without my consent.

In the consultation, he tried to give me the schpiel, including the 'it stays local' thing. The Heidi AI website has the scripts for clinicians; he ran through them all.

Oh, their documents for clinicians also mention every two sentences that patient/client consent is not required at all. I wonder why they keep saying that? Hmm.

This doctor knows I am a developer. When I asked him to explain what he meant by 'local data', he said the servers were in Australia. I almost flipped the desk. Aside from the fact that it is mandatory (it's the law! they do not have a choice!), it's ...kind of meaningless where the servers are, especially when he (on behalf of Heidi AI) was trying to sell it as a security or privacy feature. When I pointed that out, he just couldn't wrap his head around it. Of course he can't, he doesn't understand.

AHPRA's "Meeting your professional obligations when using Artificial Intelligence in healthcare" guideline[0] (not any kind of enforceable requirement, unfortunately) has great stuff in it. It encourages using it with the informed consent of patients. Even if my doctor read it and agreed with it, and cared about getting consent, how the hell can he inform patients sufficiently when he has absolutely no idea about, well, anything?

He keeps pushing it and asking me about whether I've changed my mind about allowing him to use it. No! He keeps asking me questions that only confirm he hasn't even done a perfunctory web search about why some people hate LLMs, especially in the context of PII and PHI.

I really do feel for clinicians, but these products are not the answer.

[0] https://www.ahpra.gov.au/Resources/Artificial-Intelligence-i...

I spec'd up an implementation of this that uses a hardware button with colors that is in reach of either party. The customer went with a different vendor based on price/"complexity"/training.

I genuinely think people “care” about it (in quotes). It’s one of the things where nobody cares unless something bad happens, and when bad thing happens, they shrug it off and forget about it a week later.

I’d go as far as saying she’s right. And we’re in a tiny minority for even thinking about it.

The New Zealand Chief Digital Officer allowed Australian cloud providers to be used as there weren't suitable NZ data centers and this was many years ago.

Didn't Health NZ just suffer a major data breach and have patient records ransomed?

ASR models can output a confidence score along with the text, but it is rarely used in the UI to display the results.. or maybe lost entirely in a subsequent LLM layer.

Physicians need to have it pounded into them that every hallucination is downstream harm. AI has no place in medicine. If they insist on it, then all transcripts must be stored with the raw audio. Which should be accessible side by side, with lines of transcript time coded. It's the only way to actually use these safely, while guarding against hallucinations.

This is where I'm at as a GP. Every few months I give Heidi another try, but I haven't noticed any real improvement over the last two years. It spends lots of words on trivial nonsense and misses clinically significant points and sometimes entire issues. It takes far more time to review and fix the notes than it saves in typing. Presumably it will be good enough one day, but it's not there yet.

It's Gell-Mann amnesia: you notice the errors in fields where you're an expert.

https://en.wiktionary.org/wiki/Gell-Mann_Amnesia_effect

>I think there is a chance that these systems will lead to a change where the note isn't the fundamental record of the encounter. Instead different artifacts are created specifically for each entity that needs it. Billing gets their view, and scheduling gets theirs, and, etc etc..

This is what an EHR does somewhat. The discrete data elements in the DB and the way they are displayed in the system are a better record than free text notes.

The problem is creating standards so this data is easily exchanged. Anyone can read and parse a free text note - but if we had standards this would be less necessary.

This will always happen as long as there is a combative relationship between private insurers and providers over reimbursements. Each side is using documentation or lack thereof to make their financial case.

The one my doctor was using got my obs numbers completely wrong.

We had to correct them at the end of the consultation.

that's mentioned in OP article.

Heidi is frustratingly consistent at hallucinating stuff. I've seen it in almost all of the dozen or so summaries I've had from medical people recently (surgeon, physio, consultant). A GP I know tried for a month and then was like 'it's not worth the risk exposure to me or my patients'.

This is a ridiculous sentence. Of course inference-only AI forgets. You can literally just program it that way.

In fact, it's human transcribers who chose whether to forget the details of a case or whether to share the details of an especially funny patient with their buddies at the bar.

Noticing that people in the west would rather speak about privacy than more efficient or cheaper healthcare.

Enterprises are ok sharing their code base with OpenAI. I think it should be okay for patients.