Google Lumiere Multimodal AI Video Generation Tool Unveiled; Can Create 5-Second Videos From Text, Images

Google Lumiere supports text-to-video and image-to-video models and has options to create stylised videos.

Written by Akash Dutta, Edited by Siddharth Suvarna | Updated: 29 January 2024 14:53 IST

Google Lumiere Multimodal AI Video Generation Tool Unveiled; Can Create 5-Second Videos From Text, Images

Photo Credit: Inbar Mosseri/Google

Google Lumiere is currently not available to the public

Highlights

Google Lumiere uses the Space-Time-U-Net diffusion model
It generates 80 frames for the 5-second-long video
Lumiere joins existing AI video models by Runway and Pika

Google unveiled its latest artificial intelligence (AI) model, Lumiere, last week. The new AI model is a multimodal video generation tool that can generate 5-second-long videos. It supports both text-to-video and image-to-video generation and joins existing AI models such as Runway Gen-2 and Pika 1.0. As per Google, Lumiere uses a Space-Time U-Net (STUNet) architecture that innovates how motion occurs in an AI video, making it appear realistic. The platform is not open to the public as of yet.

In an accompanying preprint paper, the research team behind Lumiere explained that the major innovation in motion comes from creating the video in a single process instead of putting together still frames. Due to this, both the spatial (the objects in the video) and temporal (how things move around in the video) aspects of the video generation are created simultaneously. For the layperson, this results in perceiving motions as they occur in nature. To achieve this, Lumiere generates a larger number of 80 frames instead of Stable Diffusion's 25 frames.

“By deploying both spatial and (importantly) temporal down- and up-sampling and leveraging a pre-trained text-to-image diffusion model, our model learns to directly generate a full-frame-rate, low-resolution video by processing it in multiple space-time scales,” the paper added.

[Sponsored] Samsung Galaxy S24: The Ultimate Camera Phone With Smarter On-Device AI

While Google Lumiere cannot be tested at the moment, the website is live and enthusiasts can check various videos created using the AI model as well as the text prompt and input images used to create the output. It can also generate videos in various styles, cinemagraphs that let users animate a certain part of the video, and inpainting where a masked-out video or image is used and the AI completes it based on the prompt.

Google's latest AI video generation tool competes with existing AI models such as Runway Gen-2, which was launched in March 2023, and Pika Lab's Pika 1.0, both of which are accessible to the public. While Pika can create 3-second-long videos (which can be increased for 4 more seconds), Runway can generate videos as long as 4 seconds. Both models are multimodal and allow video editing as well.

Is the Samsung Galaxy Z Flip 5 the best foldable phone you can buy in India right now? We discuss the company's new clamshell-style foldable handset on the latest episode of Orbital, the Gadgets 360 podcast. Orbital is available on Spotify, Gaana, JioSaavn, Google Podcasts, Apple Podcasts, Amazon Music and wherever you get your podcasts.

Affiliate links may be automatically generated - see our ethics statement for details.

Comments

For the latest tech news and reviews, follow Gadgets 360 on X, Facebook, WhatsApp, Threads and Google News. For the latest videos on gadgets and tech, subscribe to our YouTube channel. If you want to know everything about top influencers, follow our in-house Who'sThat360 on Instagram and YouTube.

Further reading: Google, Artificial intelligence

Akash Dutta Email Akash Dutta

Akash Dutta is a Senior Sub Editor at Gadgets 360. He is particularly interested in the social impact of technological developments and loves reading about emerging fields such as AI, metaverse, and fediverse. In his free time, he can be seen supporting his favourite football club - Chelsea, watching movies and anime, and sharing passionate opinions on food. More