Google DeepMind Is Integrating Gemini 1.5 Pro in Robots That Can Navigate Real-World Environments

Google DeepMind shared a video demonstration of a Gemini AI integrated robot that can guide users to the desired destination.

Written by Akash Dutta, Edited by Siddharth Suvarna | Updated: 12 July 2024 15:33 IST

Google DeepMind Is Integrating Gemini 1.5 Pro in Robots That Can Navigate Real-World Environments

Photo Credit: Google DeepMind

DeepMind said that the Gemini-integrated robots were able to perform Multimodal Instruction Navigation

Highlights

Google DeepMind has published a paper sharing impact of Gemini in robots
The robots were leveraging Gemini 1.5 Pro’s 2 million context window
DeepMind also used its Robotic Transformer 2 (RT-2) AI model

Google DeepMind shared new advancements made in the field of robotics and vision language models (VLMs) on Thursday. The artificial intelligence (AI) research division of the tech giant has been working with advanced vision models to develop new capabilities in robots. In a new study, DeepMind highlighted that using Gemini 1.5 Pro and its long context window has now enabled the division to make breakthroughs in navigation and real-world understanding of its robots. Earlier this year, Nvidia also unveiled new AI technology that powers advanced capabilities in humanoid robots.

Google DeepMind Uses Gemini AI to Improve Robots

In a post on X (formerly known as Twitter), Google DeepMind revealed that it has been training its robots using Gemini 1.5 Pro's 2 million token context window. Context windows can be understood as the window of knowledge visible to an AI model, using which it processes tangential information around the queried topic.

For instance, if a user asks an AI model about “most popular ice cream flavours”, the AI model will check the keyword ice cream and flavours to find information to that question. If this information window is too small, then the AI will only be able to respond with the names of different ice cream flavours. However, if it is larger, the AI will also be able to see the number of articles about each ice cream flavour to find which has been mentioned the most and deduce the “popularity factor”.

DeepMind is taking advantage of this long context window to train its robots in real-world environments. The division aims to see if the robot can remember the details of an environment and assist users when asked about the environment with contextual or vague terms. In a video shared on Instagram, the AI division showcased that a robot was able to guide a user to a whiteboard when he asked it for a place where he could draw.

View this post on Instagram

A post shared by Google DeepMind (@googledeepmind)

“Powered with 1.5 Pro's 1 million token context length, our robots can use human instructions, video tours, and common sense reasoning to successfully find their way around a space,” Google DeepMind stated in a post.

In a study published on arXiv (a non-peer-reviewed online journal), DeepMind explained the technology behind the breakthrough. In addition to Gemini, it is also using its own Robotic Transformer 2 (RT-2) model. It is a vision-language-action (VLA) model that learns from both web and robotics data. It utilises computer vision to process real-world environments and use that information to create datasets. This dataset can later be processed by the generative AI to break down contextual commands and produce desired outcomes.

At present, Google DeepMind is using this architecture to train its robots on a broad category known as Multimodal Instruction Navigation (MIN) which includes environment exploration and instruction-guided navigation. If the demonstration shared by the division is legitimate, this technology might further advance robotics.

Comments

For the latest tech news and reviews, follow Gadgets 360 on X, Facebook, WhatsApp, Threads and Google News. For the latest videos on gadgets and tech, subscribe to our YouTube channel. If you want to know everything about top influencers, follow our in-house Who'sThat360 on Instagram and YouTube.

Further reading: Google DeepMind, DeepMind, Gemini, AI, Artificial Intelligence, Robotics, Robots

Akash Dutta Email Akash Dutta

Akash Dutta is a Senior Sub Editor at Gadgets 360. He is particularly interested in the social impact of technological developments and loves reading about emerging fields such as AI, metaverse, and fediverse. In his free time, he can be seen supporting his favourite football club - Chelsea, watching movies and anime, and sharing passionate opinions on food. More