OpenAI Warns ChatGPT Voice Mode Users Might End Up Forming ‘Social Relationships’ With the AI

OpenAI found that some users were forming connections with ChatGPT’s Voice Mode during its early testing.

Written by Akash Dutta, Edited by Siddharth Suvarna | Updated: 9 August 2024 13:38 IST

OpenAI Warns ChatGPT Voice Mode Users Might End Up Forming ‘Social Relationships’ With the AI

Photo Credit: Pexels/Pavel Danilyuk

Another concern highlighted by OpenAI includes the AI Voice Mode influencing social norms

Highlights

OpenAI said impact of misinformation can also increase with Voice Mode
The Voice Mode might also affect healthy relationships
The AI firm said it is studying the potential for emotional reliance

OpenAI warned on Thursday that the recently released Voice Mode feature for ChatGPT might result in users forming social relationships with the artificial intelligence (AI) model. The information was part of the company's System Card for GPT-4o, which is a detailed analysis about the potential risks and possible safeguards of the AI model that the company tested and explored. Among many risks, one was the potential of people anthromorphising the chatbot and developing attachment to it. The risk was added after it noticed signs of it during early testing.

ChatGPT Voice Mode Might Make Users Attached to the AI

In a detailed technical document labelled System Card, OpenAI highlighted the societal impacts associated with GPT-4o and the new features powered by the AI model it has released so far. The AI firm highlighted that anthromorphisation, which essentially means attributing human characteristics or behaviours to non-human entities.

OpenAI raised the concern that since the Voice Mode can modulate speech and express emotions similar to a real human, it might result in users developing an attachment to it. The fears are not unfounded either. During its early testing which included red-teaming (using a group of ethical hackers to simulate attacks on the product to test vulnerabilities) and internal user testing, the company found instances where some users were forming a social relationship with the AI.

In one particular instance, it found a user expressing shared bonds and saying “This is our last day together” to the AI. OpenAI said there is a need to investigate whether these signs can develop into something more impactful over a longer period of usage.

A major concern, if the fears are true, is that the AI model might impact human-to-human interactions as people get more used to socialising with the chatbot instead. OpenAI said while this might benefit lonely individuals, it can negatively impact healthy relationships.

Another issue is that extended AI-human interactions can influence social norms. Highlighting this, OpenAI gave the example that with ChatGPT, users can interrupt the AI any time and “take the mic”, which is anti-normative behaviour when it comes to human-to-human interactions.

Further, there are wider implications of humans forging bonds with AI. One such issue is persuasiveness. While OpenAI found that the persuasion score of the models were not high enough to be concerning, this can change if the user begins to trust the AI.

At the moment, the AI firm has no solution for this but plans to observe the development further. “We intend to further study the potential for emotional reliance, and ways in which deeper integration of our model's and systems' many features with the audio modality may drive behavior,” said OpenAI.

Comments

For the latest tech news and reviews, follow Gadgets 360 on X, Facebook, WhatsApp, Threads and Google News. For the latest videos on gadgets and tech, subscribe to our YouTube channel. If you want to know everything about top influencers, follow our in-house Who'sThat360 on Instagram and YouTube.

Further reading: OpenAI, ChatGPT, ChatGPT Voice Mode, GPT, AI, Artificial Intelligence

Akash Dutta Email Akash Dutta

Akash Dutta is a Senior Sub Editor at Gadgets 360. He is particularly interested in the social impact of technological developments and loves reading about emerging fields such as AI, metaverse, and fediverse. In his free time, he can be seen supporting his favourite football club - Chelsea, watching movies and anime, and sharing passionate opinions on food. More