Home
Science
Science Features
This Is Where the Real Action in Artificial Intelligence Takes Place

This Is Where the Real Action in Artificial Intelligence Takes Place

By Elizabeth Dwoskin, The Washington Post | Updated: 10 June 2016 10:26 IST

Swarms of journalists lined the halls of a Southern California oceanfront resort recently to see tech luminaries like Jeff Bezos and Elon Musk discuss the Gawker case, rocket launches to space, and, Silicon Valley's obsession du jour, artificial intelligence.

But perhaps the real action in artificial intelligence is happening at the Hilton San Francisco, where a much less flashy tech crowd gathered this week.

There, hundreds of engineers (2,500 to be precise), geeked out over fireside chats on topics such as "streaming data in real time" and "building accelerators in databases." The event, called the Spark Summit, brings together the growing number of developers who are working with Spark, perhaps the hottest data-mining software today.

One way to understand future of artificial intelligence and how it will transform the economy is to watch what everyone is talking about at the Spark Summit. Not that a layperson could comprehend most of these talks. Engineers from companies such as Capital One to Airbnb spoke about their successful big data projects in pure engineering-ese.

But the fact that the keynotes were from revered artificial intelligence researchers Andrew Ng and Jeff Dean speak to the value of Spark. Companies such as Siemens say they have used to it to analyze data streaming in from sensors to predict which wind turbines will break. NBC Universal is collecting data on consumer behavior and using algorithms, crunched in Spark, to predict which television shows will resonate with consumers.

Some of this stuff may not sound entirely new. But it's what is happening on the backend that has gotten geeks and investors excited.

In the last four years, Silicon Valley's obsession with big data and data mining has ballooned into a full-blown artificial intelligence craze. Look at the travel industry. Today, companies like Expedia want to collect your travel interests and history, crunch the data in an app or some other platform, and build virtual travel agents who will assist you on your flight using voice recognition technology. This shift, in which automation and algorithms move into every industry, is already underway. It is powered by a growing number of backend technologies and startups aiming to win lucrative contacts with corporate America, whose names most consumers have never heard of; At least one of them may become the next Microsoft, Oracle, or IBM.

The most prominent recent example of these hyped backend big data technologies is called Hadoop. The software, named after a toy elephant owned by the son of the Yahoo programmer who helped develop it, arose in the mid-2000s when Google and Yahoo were neck-in-neck in a battle to be the dominant search engine.

Hadoop was used by both companies to index the massive amounts of data consumers were throwing up on the "Web." By 2008, investors thought the data-mining software was ripe for corporate America, and armies of salespeople began pitching it as an all-purpose big data analytics tool. They said, for example, that it could help credit card companies and banks detect fraud and other anomalies in massive amounts of financial transactions, or e-commerce companies to build recommendation features on their websites that say "people who shopped for this also shopped for this."

By 2014, investors had poured over $2 billion into dozens of companies selling services based on Hadoop. Because Hadoop is open-source, which means the core software is free for developers to share and modify, companies make money off selling tech support and other services on top of Hadoop. But Hadoop ran into trouble because the clunky tech didn't always deliver on its promises. Four years into the big data craze, the majority of big data projects fail or are never completed, according to the tech firm InfoChimps.

Now, after years of hype and investment in Hadoop, Spark may be replacing it.

Spark enables the processing of more advanced machine learning algorithms, and performs more efficiently than parts of Hadoop, and is more easily operated in the cloud, engineers say. That's unwelcome news for proponents of Hadoop, and reflects just how quickly the latest innovative technology can become passe. Of course, the same outcome could happen to Spark too.

The Spark project, which is also open source, originated in a computer science lab at the University of California. The venture capitalist Ben Horowitz discovered the technology via a tip from a professor in the department, who called to tell Horowitz that Spark's creator was the most talented computer scientist to come out of Berkeley in a decade. Almost immediately, Horowitz leapt to help the Spark engineers, most who whom were graduate students who had never run a business, to build a company. Through his venture firm Andreessen Horowitz, Horowitz and other investors have put $47 million into the company, called Databricks.

Forty-eight percent of Spark developers say they no longer use Hadoop, Databricks says, even though the project was initially considered a way to supplement it. IBM has recently announced the company is dedicating 3,500 developers to Spark-related projects.

The success of Spark means complicated, interesting things for the future of enterprise software. It's a less sexy corner of the tech world - this isn't ride-sharing or social media - but it is still where the money is.

But Spark is open source, which means it's free. Microsoft became one of the wealthiest companies in history selling proprietary software. Databricks and others still have to find a way to make real revenue by selling technology that nobody owns. As Peter Levine, another Andreessen partner who invests in open-source companies is fond of saying, open source companies have never been worth a fraction of their proprietary counterparts. Open source startups like Databricks are trying to make money selling specialized services and proprietary products that hook into and augment the open source software, including tech support and custom security products, to big corporations. But they have a long way to go.

Moreover, the creators of Spark and other artificial intelligence technologies appear to think about the implications of what they are building in a different way than Microsoft or Oracle did in their heydays. Those legacy giants wanted to capture the wealth of corporate America. Surely, today's enterprise startups also aspire to that, but AI entrepreneurs today are also preoccupied with a different set of questions - about the kind of future more artificial intelligence will bring.

As businesses become faster and more efficient, millions of jobs may disappear. Elon Musk and Silicon Valley startup incubator Y-Combinator recently dedicated a billion dollars to create OpenAI, a non-profit organization dedicated to creating powerful open-source artificial intelligence that will serve the public good. The current debate within the AI community, said Databricks chief executive Ali Ghodsi, is whether the changes wrought by AI will free people up to spend time on creative pursuits, resulting a net gain for humanity, or strip them of their means.

Comments

For the latest tech news and reviews, follow Gadgets 360 on X, Facebook, WhatsApp, Threads and Google News. For the latest videos on gadgets and tech, subscribe to our YouTube channel. If you want to know everything about top influencers, follow our in-house Who'sThat360 on Instagram and YouTube.