AI Model GPT-3 Found to Reason as Well as College Undergraduate Students

"Surprisingly, not only did GPT-3 do about as well as humans but it made similar mistakes as well," researchers said.

By Press Trust of India | Updated: 2 August 2023 18:02 IST

AI Model GPT-3 Found to Reason as Well as College Undergraduate Students

Photo Credit: Reuters

The AI tool was found to perform better than the humans' average score in SAT analogies

Highlights

GPT-3 was asked to solve reasoning problems typical tests such as the SAT
Scientists asked 40 UCLA students to solve the same problems
AI did worse than students in solving analogies based on short stories

GPT-3, the popular AI-powered tool, was found to reason as well as college undergraduate students, scientists have found.

The artificial intelligence large language model (LLM) was asked to solve reasoning problems that were typical of intelligence tests and standardised tests such as the SAT, used by colleges and universities in the US and other countries to make admissions decisions.

The researchers from the University of California - Los Angeles (UCLA), US, asked GPT-3 to predict the next shape which followed a complicated arrangement of shapes. They also asked the AI to answer SAT analogy questions, all the while ensuring that the AI would have never encountered these questions before.

[Sponsored] Samsung Galaxy S24: The Ultimate Camera Phone With Smarter On-Device AI

They also asked 40 UCLA undergraduate students to solve the same problems.

In the shape prediction test, GPT-3 was seen to solve 80 percent of the problems correctly, between the humans' average score of just below 60 percent and their highest scores.

"Surprisingly, not only did GPT-3 do about as well as humans but it made similar mistakes as well," said UCLA psychology professor Hongjing Lu, senior author of the study published in the journal Nature Human Behaviour.

Microsoft Testing Bing AI Chat on Chrome, Safari for Select Users: Report

In solving SAT analogies, the AI tool was found to perform better than the humans' average score. Analogical reasoning is solving never-encountered problems by comparing them to familiar ones and extending those solutions to the new ones.

The questions asked test-takers to select pairs of words that share the same type of relationships. For example, in the problem "'Love' is to 'hate' as 'rich' is to which word?," the solution would be "poor".

However, in solving analogies based on short stories, the AI did less well than students. These problems involved reading one passage and then identifying a different story that conveyed the same meaning.

"Language learning models are just trying to do word prediction so we're surprised they can do reasoning," Lu said. "Over the past two years, the technology has taken a big jump from its previous incarnations." Without access to GPT-3's inner workings, guarded by its creator, OpenAI, the researchers said they were not sure how its reasoning abilities worked, that whether LLMs are actually beginning to "think" like humans or are doing something entirely different that merely mimics human thought.

This, they said, they hope to explore.

"GPT-3 might be kind of thinking like a human. But on the other hand, people did not learn by ingesting the entire internet, so the training method is completely different.

"We'd like to know if it's really doing it the way people do, or if it's something brand new - a real artificial intelligence - which would be amazing in its own right," said UCLA psychology professor Keith Holyoak, a co-author of the study.

Samsung launched the Galaxy Z Fold 5 and Galaxy Z Flip 5 alongside the Galaxy Tab S9 series and Galaxy Watch 6 series at its first Galaxy Unpacked event in South Korea. We discuss the company's new devices and more on the latest episode of Orbital, the Gadgets 360 podcast. Orbital is available on Spotify, Gaana, JioSaavn, Google Podcasts, Apple Podcasts, Amazon Music and wherever you get your podcasts.

Affiliate links may be automatically generated - see our ethics statement for details.

Comments

For the latest tech news and reviews, follow Gadgets 360 on X, Facebook, WhatsApp, Threads and Google News. For the latest videos on gadgets and tech, subscribe to our YouTube channel. If you want to know everything about top influencers, follow our in-house Who'sThat360 on Instagram and YouTube.