OpenAI said impact of misinformation can also increase with Voice Mode
The Voice Mode might also affect healthy relationships
The AI firm said it is studying the potential for emotional reliance
Another concern highlighted by OpenAI includes the AI Voice Mode influencing social norms
Photo Credit: Pexels/Pavel Danilyuk
OpenAI warned on Thursday that the recently released Voice Mode feature for ChatGPT might result in users forming social relationships with the artificial intelligence (AI) model. The information was part of the company's System Card for GPT-4o, which is a detailed analysis about the potential risks and possible safeguards of the AI model that the company tested and explored. Among many risks, one was the potential of people anthromorphising the chatbot and developing attachment to it. The risk was added after it noticed signs of it during early testing.
ChatGPT Voice Mode Might Make Users Attached to the AI
In a detailed technical document labelled System Card, OpenAI highlighted the societal impacts associated with GPT-4o and the new features powered by the AI model it has released so far. The AI firm highlighted that anthromorphisation, which essentially means attributing human characteristics or behaviours to non-human entities.
OpenAI raised the concern that since the Voice Mode can modulate speech and express emotions similar to a real human, it might result in users developing an attachment to it. The fears are not unfounded either. During its early testing which included red-teaming (using a group of ethical hackers to simulate attacks on the product to test vulnerabilities) and internal user testing, the company found instances where some users were forming a social relationship with the AI.
In one particular instance, it found a user expressing shared bonds and saying “This is our last day together” to the AI. OpenAI said there is a need to investigate whether these signs can develop into something more impactful over a longer period of usage.
A major concern, if the fears are true, is that the AI model might impact human-to-human interactions as people get more used to socialising with the chatbot instead. OpenAI said while this might benefit lonely individuals, it can negatively impact healthy relationships.
Another issue is that extended AI-human interactions can influence social norms. Highlighting this, OpenAI gave the example that with ChatGPT, users can interrupt the AI any time and “take the mic”, which is anti-normative behaviour when it comes to human-to-human interactions.
Further, there are wider implications of humans forging bonds with AI. One such issue is persuasiveness. While OpenAI found that the persuasion score of the models were not high enough to be concerning, this can change if the user begins to trust the AI.
At the moment, the AI firm has no solution for this but plans to observe the development further. “We intend to further study the potential for emotional reliance, and ways in which deeper integration of our model's and systems' many features with the audio modality may drive behavior,” said OpenAI.
OpenAI Introduces New Audio Models in API, Can Be Used for Agentic Workflows21 March 2025
Search
![Gadgets 360 With Technical Guruji: Ask TG [ March 22, 2025]](https://c.ndtvimg.com/2024-12/5s52e4k_-ask-tg_640x480_14_December_24.jpg?downsize=180:*)
