Facebook's mobile apps no longer run purely on web technology. Instead, they run as native software loaded onto iPhones, iPads, Android devices and other mobile gear. The upside is that the app is much faster and more responsive than before. The downside, for Facebook, is that testing the app is far more difficult. The iPhone app is separate from the Android app, and so on.
But as the company detailed in a blog post published this morning, it has built a new tool called Airlock to help ease the pain of testing its various pieces of mobile software with real-world users. It's part of a growing number of tools that handle what's called A/B testing on mobile phones.
The technique is similar to focus group testing. Facebook designers and developers make changes to the app -- some small (like the use of different colors or icons) and some big (like a completely different navigation scheme) -- and then roll those variations out to only a small fraction of the people using the thing in the real world. The changes that work best can later be rolled out to everyone.
It's relatively easy to do A/B testing with web applications -- changes can be delivered straight from Facebook's servers to your browser -- but it can be much harder to do this with native apps. You'd have to submit each variation of an app to each app store. Then you'd have to wait for Apple and Google to approve each variation. That isn't practical for a company like Facebook that conducts A/B tests on such an enormous scale. Plus, there's no way to tell the app stores to distribute a new version of the app to, say, only 10 percent of users.
That put Facebook's developers in a bind. "Testing is an important part of our development, and we run hundreds of tests on Facebook -- most of which are rolled out to a sample of people to test their impact," Facebook developers Ari Grant and Kang Zhang wrote in their blog post. "Not every test makes it into production, but even failed tests help us understand how to improve. Losing some of this ability became a challenge."
To solve the problem, Facebook built Airlock. The developers are able to bundle all the different variations they'd like to test into a single version of the app. Once a user downloads the app from the app store, the Facebook server tells the app which variation to show.
Facebook started off by using Airlock to test variations on employees' phones, but later graduated to using it to test across all its users, a Facebook spokeperson tells us. One of the biggest features tested with Airlock thus far is the new navigation pane that appears in the Android and iPhone versions of the Facebook app.
"This tool has allowed us to now scale the framework to support 10 or 15 different variations of a single experiment and put it in the hands of millions of people using our apps," the developers write.
Facebook is keeping Airlock to itself for now, though a spokesperson says the company is evaluating whether it will provide value to the open source community. In the meantime, a team of ex-Googlers has released a mobile A/B testing platform called Leanplum. But a little open source competition would be a good thing.
