Facebook Manages to Squeeze an AI Into Its Mobile App

The new Facebook app will run deep neural networks on your phone, helping to open the door for the next wave in AI.
Image may contain Human Person Electronics Screen Monitor Display Vehicle Transportation Bicycle and Bike
Facebook

Hussein Mehanna is showing off a new incarnation of the Facebook smartphone app. It can transform a photo of your backyard barbecue into a Picasso. Or a Van Gogh. Or a Warhol.

The app includes a particularly extravagant photo filter. You select a work of art---something akin to, say, a 1907 Picasso---and it creates a Cubist incarnation of your backyard barbecue. It's fun, and it works even with live video. Turn the camera on yourself, and you too can be a Picasso. But that's not half as interesting as the technology that underpins Facebook's new app and its extravagant photo filter. Mehanna is one of the Facebook engineers working to push artificial intelligence across the company, and as he explains, the app includes several deep neural networks, a form of artificial intelligence that's rapidly reinventing the tech world.

Based loosely on the web of neurons in the human brain, neural networks can learn discrete tasks by analyzing vast amounts of data. This is what identifies faces in the photos you post to Facebook, recognizes the commands you speak into your Android phone, and helps translate your Skype calls into foreign languages. Now, using various works of art, Facebook is training neural networks to inject a new look into your personal pics.

Typically, neural networks run on large numbers of computer servers packed into data centers on the other side of the Internet---they don't work unless your phone is online---but with its new app, Facebook takes a different approach. The Picasso filter is driven by a neural network efficient enough to run on the phone itself. "We perceive the world in real-time," Mehanna says. "Why wouldn't you want the same thing from your AI?"

Already available in Ireland and due soon here in the States, this new Facebook app is another sign that deep neural networks will push beyond the data center and onto phones, cameras, and various other devices spread across the so-called Internet of Things. Last summer, Google squeezed a neural network into its Google Translate app, which can identify words in photos and translate them in new languages. And so many other operations, including the Allen Institute for Artificial Intelligence, are developing similarly svelte neural networks.

Yes, these tools can operate without an Internet connection. And that points to a future where our smartphone apps can perform a much wider range of tasks while offline. But it also shows we're moving towards technology that can handle more complex AI tasks with less delay. Ultimately, if you can complete a task without sending a bunch of data across the wire, it will happen quicker.

Imagine apps that can instantly recognize faces or objects when you point your phone at them. Think what this could do for people who are blind or otherwise visually impaired. "Doing this on the phone changes the nature of the game," says Allen Institute CEO Oren Etzioni, pointing out that this can even help drive augmented reality headsets like the Microsoft Hololens. If a device can more accurately recognize the world around it, it can more accurately augment that reality.

Justin Johnson, Alexandre Alahi, and Li Fei-Fei
Training Versus Execution

A neural network operates in two stages. First, a company like Facebook or Google trains it for a particular task, like image recognition or machine translation. Facebook might teach a neural network to recognize goats, for instance, by feeding it millions of goat photos. Then someone like you or me executes the neural network. We give it a photo, and it tells us whether the photo includes a goat.

Facebook's app doesn't train its neural networks on your smartphone. That still happens on servers in the data center. But the phone does execute the neural net---without calling back to the data enter. That may seem like a small thing, but building a deep neural net that can so quickly execute on a phone---which offers limited processing power and memory---is no simple task. The new photo filter is based on neural network technology first described by a team of German researchers in 2015, and that technology couldn't operate in real-time, even though it ran on data center hardware. Little more than a year later, Facebook is doing pretty much the same thing on a phone---without delay. For Facebook chief technology officer Mike Schroepfer, this shows just how fast AI is evolving.

Part of the trick is that Facebook has minimized the complexity of neural network that transforms your photo into a Picasso---something similar to Google's approach with its Translate app. The training stage still takes an awfully long time: According to Facebook engineering director Tommer Leyvand, the neural net must train for a good 400 hours on GPU chips, the processors typically used for AI training inside the data center. Basically, after training a neural net to recognize objects in photos, the team feeds it a famous work of art, retraining it to apply the same style to those objects. But in the end, Menhana and team honed this neural net so that it only uses the most important parts of what it learned.

At the same time, the team built a new piece of software designed specifically for executing neural networks with the limited resources available on mobile phones. This AI framework is called Caffe2Go, and according to Facebook, it can execute neural nets in less than 1/20th of a second. Naturally, execution times depend on what models are being executing. But the larger point is that Facebook intends to offer the framework on both iOS and Android devices, intent on building all sorts of AI models that can operate without a tether to the data center. "With anything we can build on the server, we now have a vehicle to ship it on mobile devices---and soon," Schroepfer explains. He says that Facebook is already experimenting with mobile neural networks that can recognize objects in videos at 60 frames a second.

Beyond Picasso

Eventually, this kind of work will create a virtuous circle of AI evolution. As companies like Facebook and Google continue to push neural networks onto smartphones, phone makers will start building hardware into these devices that can run neural networks with even greater speed. That, in turn, will yield even more complex apps. And so on. Schroepfer says Facebook is already talking to the major mobile chip makers about modifying their processors for use with future AI.

Meanwhile, some companies are building entirely new processors that could accelerate the execution of neural networks on phones and other devices. This includes Movidius, a company recently acquired by Intel, the world's largest chip maker, as well as IBM. And if these chips work as advertised, they will find a home in the market. "The demand will be there," Schroepfer says.

A Picasso photo filter won't change your life. But this one points to big changes in the years to come.