OpenAI’s Transcription Tool Reportedly Adding Hallucinated Content in Medical Consultation Records

OpenAI’s Whisper transcription tool is reportedly adding racial information and imaginary medication in consultation records.

Written by Akash Dutta, Edited by Siddharth Suvarna | Updated: 28 October 2024 16:44 IST

OpenAI’s Transcription Tool Reportedly Adding Hallucinated Content in Medical Consultation Records

Photo Credit: Unsplash/Levart_Photographer

OpenAI claims Whisper comes with “human-level robustness and accuracy”

Highlights

OpenAI’s Whisper-based tools are reportedly being used in US hospitals
OpenAI released the transcription tool in 2022
Every 8 out of 10 Whisper transcriptions reportedly contain hallucination

OpenAI released an artificial intelligence (AI) tool dubbed Whisper in 2022, which can transcribe speech to text. However, a report claimed that the AI tool is prone to hallucinations and is adding imaginary text in transcriptions. This is concerning as the tool is said to be used in several high-risk industries such as medicine and accessibility. A particular concern reportedly comes from the use of the tool in doctor-patient consultations, where hallucination can add potentially harmful information and put the patient's life at risk.

OpenAI Whisper Reportedly Prone to Hallucinations

The Associated Press reported that OpenAI's automatic speech recognition (ASR) system Whisper has a high potential of generating hallucinated text. Citing interviews with multiple software engineers, developers, and academic researchers, the publication claimed that the imaginary text includes racial descriptions, violence, and medical treatments and medications.

Hallucination, in the AI parlance, is a major issue which causes AI systems to generate responses which are incorrect or misleading. In the case of Whisper, the AI is said to be inventing text which was never spoken by anyone.

[Sponsored] Samsung Galaxy S24: The Ultimate Camera Phone With Smarter On-Device AI

In an example verified by the publication, the speaker's sentence, “He, the boy, was going to, I'm not sure exactly, take the umbrella.” was changed to “He took a big piece of a cross, a teeny, small piece ... I'm sure he didn't have a terror knife so he killed a number of people.” In another instance, Whisper reportedly added racial information without any mention of it.

While hallucination is not a new problem in the AI space, this particular tool's issue is more impactful as the open-source technology is being used by several tools that are being used in high-risk industries. Paris-based Nabla, for instance, has created a Whisper-based tool which is reportedly being used by more than 30,000 clinicians and 40 health systems.

Nabla's tool has been used to transcribe more than seven million medical visits. To maintain data security, the company also deletes the original recording from its servers. This means if any hallucinated text was generated in these seven million transcriptions, it is impossible to verify and correct them.

Microsoft, OpenAI Spending Millions to Help News Outlets Adapt to AI

Another area where the technology is being used is in creating accessibility tools for the deaf and hard-of-hearing community, where again, verifying the accuracy of the tool is significantly difficult. Most of the hallucination is said to be generated from background noises, abrupt pauses, and other environmental sounds.

The extent of the issue is also concerning. Citing a researcher, the publication claimed that eight out of every 10 audio transcriptions were found to contain hallucinated text. A developer told the publication that hallucination occurred in “every one of the 26,000 transcripts he created with Whisper.”

OpenAI Says ChatGPT Does Not Add Bias Based on Identity of Users

Notably, at the launch of Whisper, OpenAI said that Whisper offers human-level robustness to accents, background noise, and technical language. A company spokesperson told the publication that the AI firm continuously studies ways to reduce hallucinations and has promised to incorporate the feedback in future model updates.

Comments

For the latest tech news and reviews, follow Gadgets 360 on X, Facebook, WhatsApp, Threads and Google News. For the latest videos on gadgets and tech, subscribe to our YouTube channel. If you want to know everything about top influencers, follow our in-house Who'sThat360 on Instagram and YouTube.

Further reading: OpenAI, OpenAI Whisper, AI, Artificial Intelligence

Akash Dutta Email Akash Dutta

Akash Dutta is a Senior Sub Editor at Gadgets 360. He is particularly interested in the social impact of technological developments and loves reading about emerging fields such as AI, metaverse, and fediverse. In his free time, he can be seen supporting his favourite football club - Chelsea, watching movies and anime, and sharing passionate opinions on food. More