Nvidia Debuts Fugatto AI Model That Can Generate Music, Voices and Sound Effects

Nvidia’s Fugatto is short for Foundational Generative Audio Transformer Opus 1.

Written by Akash Dutta, Edited by Siddharth Suvarna | Updated: 26 November 2024 14:26 IST

Nvidia Debuts Fugatto AI Model That Can Generate Music, Voices and Sound Effects

Photo Credit: Reuters

The full version of Fugatto uses 2.5 billion parameters and was trained on Nvidia DGX systems’ datasets

Highlights

The AI audio model accepts a combination of text and audio as input
Nvidia’s Fugatto can remove or add instruments from an existing song
Nvidia says Fugatto has the ability to combine free-form instructions

Nvidia introduced a new artificial intelligence (AI) model on Monday that can generate a variety of audio and mix different types of sounds. The tech giant calls the foundation model Fugatto, which is short for Foundational Generative Audio Transformer Opus 1. While audio-focused AI platforms such as Beatoven and Suno exist, the company highlighted that Fugatto offers users granular control over the desired output. The AI model can generate or transform any mix of music, voices and sound based on specific prompts.

Nvidia Introduces AI Audio Model Fugatto

In a blog post, the tech giant detailed its new large language model (LLM). Nvidia said Fugatto can generate music snippets, remove or add instruments from an existing song, change accent or emotion in a voice, and “even let people produce sounds never heard before.”

The AI model accepts both text and audio files as input, and users can combine both to fine-tune their requests. Under the hood, the foundation model's architecture is based on the company's previous work in speech modelling, audio vocoding, and audio understanding. Its full version uses 2.5 billion parameters and was trained on the datasets of Nvidia DGX systems.

Samsung Wins Patent for a Tri-Fold Smartphone With Specialised Barrier Layer for Improved Durability

Nvidia highlighted that the team that built Fugatto collaborated from different countries globally including Brazil, China, India, Jordan, and South Korea. The collaboration of people from different ethnicities has also contributed to developing the AI model's multi-accent and multilingual capabilities, the company said.

Coming to the AI audio model's capabilities, the tech giant highlighted that it has the capability to generate audio output types that it was not pre-trained on. Highlighting an example, Nvidia said, “Fugatto can make a trumpet bark or a saxophone meow. Whatever users can describe, the model can create.”

Additionally, Fugatto can combine specific audio capabilities using a technique called ComposableART. With this, users can ask the AI model to generate an audio of a person speaking French with a sad feeling. Users can also control the degree of sorrow and the heaviness of the accent with specific instructions.

Google DeepMind’s New AI Model Can Help in Drug Discovery

Further, the foundation model can also generate audio with temporal interpolation, or sounds that change over time. For instance, users can generate the sound of a rainstorm with crescendos of thunder that fade into the distance. These soundscapes can also be experimented with, and even if it is a sound that the model has never processed before, it can create them.

OpenAI Might Be Struggling to Improve Its Next AI Model Significantly

At present, the company has not shared any plans to make the AI model available to users or enterprises.

Comments

For the latest tech news and reviews, follow Gadgets 360 on X, Facebook, WhatsApp, Threads and Google News. For the latest videos on gadgets and tech, subscribe to our YouTube channel. If you want to know everything about top influencers, follow our in-house Who'sThat360 on Instagram and YouTube.

Further reading: Nvidia, Fugatto, AI, Artificial Intelligence, AI Model, LLM

Akash Dutta Email Akash Dutta

Akash Dutta is a Senior Sub Editor at Gadgets 360. He is particularly interested in the social impact of technological developments and loves reading about emerging fields such as AI, metaverse, and fediverse. In his free time, he can be seen supporting his favourite football club - Chelsea, watching movies and anime, and sharing passionate opinions on food. More