For pianist, software is replacing sonatas

The pianist Robert Taub was puttering around the house one afternoon in 2004 while his teen-age daughter was practicing for a violin lesson -- a Schubert sonatina in A minor. His assessment of her playing was diplomatic: "She needed to be reminded about notes and rhythms."

What followed was a brainstorm that explains why Mr Taub -- who made his reputation playing two distinctly different B's, Beethoven and Milton Babbitt -- has put his performing on hold, and why "software entrepreneur" now tops his résumé.

"I thought, wouldn't it be wonderful if she could take a photograph of her page of music and hear it instantaneously," he recalled. "She'd know what the right notes are, and what the right rhythms are, and she could imitate what she heard."

Soon he was dreaming of a device -- or maybe just software running on a computer -- that could do everything he had learned to do in music theory class: read and play a printed musical score, and listen to a passage of music and transcribe it, down to the key signature, the tempo and the time signature. He said that a quick check showed that nothing then on the market could do all that.

So Mr Taub started exploring the world of machine-learning technology. Before long he had organized a startup company and was spending more time on conference calls than at the piano.

"This is what I eat, dream, sleep," he said. Now 54, he last performed in the summer of 2008 in Aspen, Colo.

The company, MuseAmi, now has half a dozen software engineers and about as many patents applied for. The company's name (pronounced myooze-ah-MEE) is a play on words: "You become the music: muse am I," he said. "But it's also a musical friend."

In recent years the world of software for musicians has exploded, with performers using their smartphones as metronomes or tuning devices. Mr  Taub has plunged in with MuseAmi's first app, Improvox, which made its debut on iTunes last month, at $7.99 a download, and has sold a few thousand so far, Mr  Taub said. It is not the photo-and-play app he had dreamed of; he said he expected to introduce that by the end of the year.

Instead, Improvox promises to do much of what a well-equipped commercial recording studio can do, correcting notes that are sharp or flat. That is a task that was pioneered by Auto-Tune software, a plug-in for audio software used in recording studios. Being able to fix pitch or intonation problems is particularly helpful after a recording session, when the engineer and the producer discover that their big-name singer was slightly off.

MuseAmi uses different technology to correct pitch, and does so in real time. (Auto-Tune works on recordings that have already been made.) It can also generate harmonies, chosen by icons on its touch-pad screen. The icon that looks like Johann Sebastian Bach  gives a singer a Baroque backup. The icon that looks like a barber pole adds three other voices for a barbershop quartet sound, with dominant chords. And there are other icons for other effects, but as Mr Taub explained, "you don't need to know any music theory."

But in developing Improvox Mr. Taub had to learn a lot about software. First he went on a manhunt for software developers. He asked around at the Institute for Advanced Study in Princeton, NJ, where he had been an artist in residence. The name that kept coming up was that of Yann LeCun, a computer science researcher who had worked for AT&T Bell Laboratories. He was known for work involving something called convolutional neural networks, which he had developed for software that banks use to scan and read checks.

Mr Taub cold-called Mr LeCun. They met in a New Jersey diner, and Mr LeCun "sketched out on a napkin how we were going to do it," Mr Taub recalled.

Mr LeCun said they focused on technologies that they believed they could develop fairly quickly. "Some of those things drew on my experience with machine learning," Mr  LeCun said. But he said he also drew on his own experience with wind instruments: he plays the recorder, the oboe and the crumhorn.

Mr Taub found financing chiefly from Bob Stockman, whose merchant banking group has primarily financed medical-device companies. They knew each other because their children had attended the same preschool and because Mr. Stockman had gone to some of Mr. Taub's concerts.

Mr Taub began hiring a team of software engineers and developing algorithms. The team members had a "Watson, come here" moment for the read-and-play function when they photographed the sheet music for the theme from "The Godfather," and the software played it back. But the record-and-play function and the pitch correction came first, and the initial reviews from musicians are positive.

"It rips the mask off the studio tricks for your everyday music fan," said Billy Mann, a record producer and songwriter who has worked with Pink, Ricky Martin  and the Backstreet Boys, among others. "If someone wants to sing and they want to know how reverb or phasing or flanging works, they can play with it. More than anything, I think it's fun for someone who's an amateur and wants to mess around with some of the high-end professional capabilities that people have in big studios, and do it on their phone."

Christianne Orto, associate dean and director of recording and distance learning at the Manhattan School of Music, said it was aimed at the "entertainment 'prosumer' market."

"It doesn't tell you you're sharp or flat" the way educational software might, it just makes the correction, she said. "And when it harmonizes, it's not giving the harmonic structure underneath. It is somewhat glorified karaoke, but it could be a wonderful portal, a great first step for giving someone some confidence in the possibility of musical accomplishment."

Mr Taub has his own sense of accomplishment. Recalling that it had taken him eight years to learn Beethoven's daunting "Hammerklavier" Sonata, he said, "This started a little less than five years ago now, and we already have some market traction."