Google Reportedly Asking Contractors to Rate Gemini Prompts Outside of Their Expertise

Google reportedly contracts Gemini’s response evaluation to GlobalLogic.

Written by Akash Dutta, Edited by Siddharth Suvarna | Updated: 19 December 2024 13:59 IST

Google Reportedly Asking Contractors to Rate Gemini Prompts Outside of Their Expertise

Photo Credit: Reuters

If true, this raises concerns over Gemini’s accuracy in answering highly technical queries

Highlights

These contractors reportedly check Gemini’s responses for truthfulness
Earlier, these contractors reportedly had the option to skip prompts
Now, Google is said to have removed the option to skip Gemini prompts

Google is reportedly asking contractors working on evaluating Gemini's responses to rate prompts outside of their domain of expertise. As per the report, the Mountain View-based tech giant has removed the option to skip prompts, which was exercised by these contractors if they felt they did not have enough knowledge about a subject to rate the response. With artificial intelligence (AI) hallucinations as a major concern for chatbots, this reported development could lead to a drop in the quality of Gemini's responses when it comes to highly technical topics.

Google Reportedly Not Letting Contractors Skip Gemini Prompts

According to a TechCrunch report, Google has sent a new internal guideline to contractors working on Gemini. Claiming to have seen the memo sent by the tech giant, the publication claims that these contractors are now being asked to answer queries even when they might not possess the knowledge to correctly assess the answers.

Google reportedly outsources the evaluation of Gemini's responses to GlobalLogic, a firm owned by Hitachi. The contractors working on Gemini are said to be tasked with reading technical prompts and rating the AI's responses based on multiple factors such as truthfulness and accuracy. These individuals evaluating the chatbot hold expertise in specific disciplines such as coding, mathematics, medicine, and more.

[Sponsored] Samsung Galaxy S24: The Ultimate Camera Phone With Smarter On-Device AI

So far, the contractors could reportedly skip certain prompts if it was outside of their domain. This ensured that only those qualified to understand and evaluate technical responses generated by Gemini were doing so. This is a standard post-training practice for foundational models and allows AI firms to ground their responses and reduce the instances of hallucination.

However, this changed when GlobalLogic reportedly announced the new guidelines last week that contractors were no longer allowed to skip prompts unless the response was “completely missing information” or it contained harmful content that requires special consent forms to evaluate.

As per the report, the new guideline states that contractors should not “skip prompts that require specialised domain knowledge” and instead, they should rate the parts of the prompt they understand. They were reportedly also asked to include a note mentioning that they do not have the domain knowledge.

One contractor stated in internal communication, “I thought the point of skipping was to increase accuracy by giving it to someone better,” the publication claimed.

Comments

For the latest tech news and reviews, follow Gadgets 360 on X, Facebook, WhatsApp, Threads and Google News. For the latest videos on gadgets and tech, subscribe to our YouTube channel. If you want to know everything about top influencers, follow our in-house Who'sThat360 on Instagram and YouTube.

Further reading: Google, Gemini, AI, Artificial Intelligence

Akash Dutta Email Akash Dutta

Akash Dutta is a Senior Sub Editor at Gadgets 360. He is particularly interested in the social impact of technological developments and loves reading about emerging fields such as AI, metaverse, and fediverse. In his free time, he can be seen supporting his favourite football club - Chelsea, watching movies and anime, and sharing passionate opinions on food. More