Microsoft Rolling Out Copilot Vision in Preview, Can Understand User’s Browsing Activity

Copilot Vision is currently available in preview to select Copilot Pro subscribers in the US.

Written by Akash Dutta, Edited by Siddharth Suvarna | Updated: 6 December 2024 18:53 IST

Microsoft Rolling Out Copilot Vision in Preview, Can Understand User’s Browsing Activity

Photo Credit: Microsoft

Microsoft says after the end of a browsing session, the information shared with Copilot is deleted

Highlights

Microsoft says Copilot Vision will work when using the Edge browser
Copilot Vision supports voice-based prompts
Microsoft has kept Copilot Vision opt-in by default

Microsoft introduced a new artificial intelligence (AI) capability for Copilot on Thursday. Dubbed Copilot Vision, it now enables the AI chatbot to see and understand the context of what a user does online. The Redmond-based tech giant said that Copilot can access the Microsoft Edge browser. The vision-enabled chatbot can now monitor which websites and web pages the user browses, and then can perform certain commands when prompted. This capability is currently available in preview to a select number of Copilot Pro subscribers in the US.

Microsoft Copilot Vision

The Windows maker has been trying to bring its AI-powered chatbot Copilot to the forefront of the PC experience. Earlier, it unveiled purpose-built AI agents to complete tasks on the PC, and now, it has added another capability to Copilot that will let it answer user queries based on their browser-related activity, as per the blog post. Notably, Copilot Vision is not an agentic function, so the AI cannot take actions on behalf of the user.

Copilot Vision can understand the full context of what a user sees on the Edge browser. It can see the images and elements, read the text, and even comprehend the navigation buttons. The vision-enabled chatbot will be present at the bottom of the screen and users can either type or speak the prompt.

Microsoft Finally Releases Controversial AI Recall Feature in Preview

Coming to what the AI can do, it can answer questions, summarise and translate text, and spotlight certain parts of the page when prompted to do so. Microsoft states that Copilot Vision can also act as a game assistant and help a user understand how to play games like GeoGuessr, and offer tips while playing a game of Chess online.

Highlighting examples, the tech giant said Copilot Vision can help users plan an outing at the museum by pointing out the important information a user should know. It can also help users with shopping by recommending the right products based on the user's needs and preferences.

On privacy and security, Microsoft stated that Copilot Vision will be entirely opt-in and users can decide when to turn it on and when to keep it turned off. The AI will not be able to see something that the user does not want to. Additionally, any conversation data with the AI will be deleted at the end of each session. However, the Copilot's responses are logged and they will be used to improve the company's safety systems.

Microsoft’s AI Agents Will Soon Complete Complex Tasks Automatically

Microsoft further added that Copilot Vision does not capture, store, or use any data from publishers to train its AI models. It is also working with a group of external testers (red teamers) to improve the safety of the new feature.

NASA and Microsoft Launch AI Tool to Make Earth Data Accessible

Copilot Vision is currently rolling out to a limited number of Copilot Pro subscribers in the US and it can be accessed via Copilot Labs. Notably, Vision will currently work only on a select set of websites, and the functionality will be expanded gradually.

Affiliate links may be automatically generated - see our ethics statement for details.

Comments

Catch the latest from the Consumer Electronics Show on Gadgets 360, at our CES 2025 hub.

Further reading: Microsoft, Microsoft Copilot, AI, Artificial Intelligence, Microsoft Edge

Akash Dutta Email Akash Dutta

Akash Dutta is a Senior Sub Editor at Gadgets 360. He is particularly interested in the social impact of technological developments and loves reading about emerging fields such as AI, metaverse, and fediverse. In his free time, he can be seen supporting his favourite football club - Chelsea, watching movies and anime, and sharing passionate opinions on food. More