Facebook Paper Has Forever Changed the Way We Build Mobile Apps

Mike Matas was sitting on an L-shaped couch inside one of the largest offices at Facebook, holding an iPhone that plugged into a Mac laptop through a long, black cord. It was the early afternoon, and he was surrounded by several Facebook colleagues, including Chris Cox, who oversees the development of new products at the social networking giant as one of the top lieutenants to CEO Mark Zuckerberg.

The office belongs to Cox, and it often hosts meetings like this one, where Matas was about to reveal something he'd hacked together earlier that morning, after the idea came to him the night before. As the others watched, Matas tapped on his iPhone and opened a digital photo of Niagara Falls. The phone zoomed in on the heart of the image, showing the glistening falls in sharp detail, and then, simply by tilting the phone back and forth, he could explore other parts of this high-res photo, panning across the image as if he was moving through a virtual world or a 3D game.

It was the birth of the "tilt-to-explore" photo viewer now offered by Facebook Paper, the new iPhone news reading app that is, in many ways, redefining the art of mobile software. "Everyone's jaws just dropped," remembers Michael Reckhow, who was sitting beside Matas that afternoon. "Everyone started exchanging these glances that were like: 'What did he just do?'"

What's more remarkable is that Mike Matas is not a software engineer. He doesn't know Objective-C, the programming language used to build iPhone apps, or any other programming language. And yet, in a matter of hours, he could build a prototype that explored photos in a way that surprised even the seasoned engineers who gathered in Chris Cox's office that afternoon.

The trick lies with a new design tool called Origami -- a tool that lets Matas rapidly prototype mobile apps without writing a single line of computer code. Together with other Facebookers, Matas created Origami by repurposing and reshaping a computer graphics program that originated at Apple. Many tools let you prototype app designs, but when you build prototypes with Origami, they work much like a real app works, and they run on real phones -- though the phones must be plugged into a high-powered PC (thus the long, black cord that Matas ran to his laptop). This is how Facebook built much of Paper. Matas and other designers used Origami to create unusually complete prototypes, and then a group of software engineers reproduced and refined these prototypes, building software they could ship to a world of phones.

The birth of the photo viewer is a prime example of how Matas and the rest of the Facebook Paper team -- a team that operates almost like a startup inside the social networking giant -- are changing the way mobile apps are built. Origami is just one of many tools the team used to bootstrap the development of Paper, and in keeping with Zuckerberg's mission to make the world "more open and connected," Facebook is sharing these tools with the world at large. It has already open-sourced some of them, including Origami, and it plans on sharing more. Launched early last month, Paper has yet to find serious traction among Facebook users, but the company's higher aim is to help promote this kind of eye-popping design -- where movements and finger gestures drive an interface that feels an awful lot like online gaming -- across the mobile landscape.

These efforts to advance the art and science of mobile software development both inside and outside the company are part of an enormous change in recent years across Silicon Valley and beyond. A world of internet companies and independent developers are now regularly open-sourcing and freely discussing the software used to build and operate even their newest products and services -- sharing everything from mobile development tools to the sweeping data center software that underpins the largest sites on the net -- and in doing so, they're pushing technology forward at a faster rate than ever before. Facebook is at the forefront of this movement, not only because Zuckerberg is so committed to the open source way, but because, like Google and Twitter and a few others, the company has the power and the money to hire some of the most talented engineers and designers on the planet.

Facebook That Isn't Facebook

At 19, after building Mac apps with a software outfit called The Omni Group while still in high school and launching his first startup, Matas was hired by Apple. There, he helped shape the user interface of the iPhone, the iPad, and the Mac, and by the age of 23, he had launched another company, Push Pop Press, together with an ex-Apple engineer named Kimon Tsinteris. He came to Facebook when the social networking giant acquired Push Pop in 2011.

Push Pop offered a service that let authors and publishers create interactive digital books for devices like the iPhone and iPad, but after the company was acquired, Matas and the rest of his team went to work on something rather different: a mobile app that aimed to significantly change the way people use Facebook. The result was Paper, a tool that combines Facebook with a digital magazine. It lets you browse Facebook status updates and photos as well as stories from across the web -- some of them curated by a staff of editors at the company.

In some ways, it was an odd play. The aim was to not to replace Facebook's existing mobile application but to offer an alternative to it -- an app that could venture down new avenues without taking away the basic Facebook experience that hundreds of millions of people are so married to. "The idea was to revisit Facebook as an experience on mobile devices -- effectively from the ground up," Scott Goodson, another ex-Apple engineer recruited to work on he project, tells WIRED one afternoon at Facebook headquarters on Menlo Park, California, wearing a t-shirt with a Venn diagram printed on the chest. It shows the intersection of "art" and "science" as "wonder."

This new mobile app became the first project developed under the aegis of something called Facebook Creative Labs. This isn't a physical lab or even a team of people. It's more of an attitude, an effort to promote projects that could take the company in new directions. "Creative Labs is not a place or a building or a team," explains Michael Reckhow, who previously worked at Microsoft and Amazon and is now the Paper product manager. "It's an identifier that we place on a project that says: 'This is something that is going to be separate, that we're going to give room to grow.'"

Paper embodies this ethos in more ways than one. On one level, Reckhow and his team were building a very new kind app, and on another, they were fashioning a new collection of design and development tools that could help bring this app to fruition. In order to build a new kind of app, you often need new tools.

When Building Software Is Like Playing Music

One of these tools is Origami. Based on an Apple graphics tool called Quartz composer, Origami is really just a way of building images. It lets Matas and other designers fashion prototypes by piecing together hundreds of tiny graphical widgets and animations. But these images can behave like complete applications. They can even tap into live data generated by smartphones, drawing on, say, the gyroscope that tracks how a phone is moving.

For Matas and others on the team, the tool feeds the creative process in a way other prototyping tools do not. It's not just that designers can build remarkably complex prototypes. They can also remake these prototypes on the fly. "You're able to see exactly what you're doing as you're building stuff," Matas says.

In the past, he says, if you really wanted to see how an idea would work, you had to actually build an application with software code -- or get someone to code it for you. This isn't always ideal, especially for non-coders like Matas. "With your typical programming language, you have to type in a bunch of code and hit 'compile,' and a minute later, you see what you built," he says. "It's almost like you're trying to learn how to play the piano and you have a piano where you hit a couple of keys and then hit a compile button and a minute later you hear what you played." Origami changes this.

"It's like you can play and hear at the same time. You can create a feedback loop where you're able to improvise and come up with ideas on the spot," Matas says of the prototyping tool. "When you come up with a new behavior, you're able to see in real time how that works." That's what happened with the photo viewer. Matas had the idea one evening at home -- after attempts to add an automatic "Ken Burns effect" to Paper failed to, well, pan out -- and the next morning, he spent a few hours mocking it up with Origami. "I really don't think he would have invented something like this," says Reckhow, "if he didn't have the tools to perceive whether it was possible."

But there's a knock-on effect as well. According to the engineers on the team, these remarkably full prototypes have a way of feeding their creative juices as well. Eighteen-year-old Grant Paul -- who joined Facebook straight our of high school after someone read a WIRED article about his work with jailbroken iPhones -- talked his way onto the Paper team after laying eyes on the initial prototypes. "It was a new vision," he says. "I thought: 'How far can you take this platform?'"

The Shuttle Bus Effect

Facebook Paper engineers like Paul are the ones who take these Origami prototypes and then work to transform them into a mobile application that work without a tether running to a laptop. "You have to turn it into a fast and usable experience that happens even while the rest of the app is doing everything else it needs to do," he says.

This requires a whole new round of creativity. And given the extreme ambition of the Origami prototypes, it requires months of trial and error. The prototypes are so complex, Scott Goodson says, that the engineering team needed new tools of its own. One of those is known as Tweaks.

Tweaks is a bit like Origami. But rather than providing a way of quickly molding prototypes, it lets engineers instantly shape and reshape an application after they've actually built it with software code. Both designers and engineers can test changes to an app without having to recode and recompile it. Instead, they can open a menu that lets them adjust all sorts of specific behavior, including the way the app's smorgasbord of interactive animations responds to movements and finger gestures. There are times when the app runs literally dozens of physics simulations that all work in concert -- Grant worked on an animation that involved 42 virtual springs -- and Tweaks provides a way of instantly changing the behavior of each and every one of these simulations.

"We can adjust how the app works -- how bouncy an animation is or how many particles explode when [an event happens]," Reckhow says. "So, we start with this interactive prototype that's very flexible and editable and iterative, and then, when we put together the actual app, it is still flexible -- with these dials you can use to change it."

Tweaks was instrumental in transforming the Paper photo viewer from a prototype into a complete application. As Ben Cunningham, another Paper engineer, explains, initial versions of the photo viewer ran into problems when they were used on the Facebok shuttle bus from San Francisco to Menlo Park and other moving vehicles, because the movement of the vehicle would effect the phone's gyroscope and accelerometer. "When we first built it, it felt pretty good in your hand," he says. "But we noticed that, the more places we took it, it began to fall apart." With Tweaks, as he rode home on the Facebook shuttle, he could instantly adjust and readjust the filters used to eliminate any irrelevant movement, identifying what worked and what didn't without having to rebuild and recompile.

"This is crucial. We're developing a system with so many moving parts and inputs. There is a ton of refinement to get it right," Cunningham says. "If something doesn't feel good, I can start tweaking it -- right on my device -- which is really, really powerful. We can continue, continue, continue to refine. Everywhere we go."

The Core Problem

But when you're building a mobile app that behaves like this, there's an added wrinkle. Paper's user interface -- which is so dependent on physics-simulating animations -- could very easily bog down the processors on today's smartphones. And if that happens, the app becomes almost unusable. In many cases, in using the app, you physically drag virtual items across you screen, and if there's even a slight delay in that drag, the interface falls apart. "It breaks the illusion," Goodson says.

The newest iPhones now include "multi-core" processors, which can execute multiple tasks in parallel. But, according to Goodson, the way that Apple's software engine distributes tasks across these cores isn't suited to the highly interactive animations that characterize Paper. "The app is much more sensitive than most to any sort of stall -- even if it's just a few milliseconds -- that could cause stutters or dropped touch input events," he says.

In essence, Paper must run all animations on the primary processor core, because they must instantly respond to user gestures, and that means shifting other tasks to secondary cores. In order to do that, Goodson and his fellow Paper engineers also built a software engine that gives them more freedom to move tasks from core to core. This engine doesn't have an official name yet, but basically, it's a layer of software that runs atop the execution engine that Apple provides. "We've built an entire architecture that allows us to implement these visually sophisticated and interactive experiences -- with great convenience," Goodson says.

This comes into play, for example, when you're scrolling through photos on the app. After moving to one photo, you can instantly move to the next, even before the first one is completely open -- an unusually nimble arrangement for an iPhone app. According to Goodson, Facebook's engine lets it instantly interrupt the decoding of one image and move to another that is already being decoded on another core. This would not have been possible, he says, with the basic engine provided by Apple.

Into the Future

The result is a news reader that feels almost like a game or a virtual world -- at least in the way you use it. "There's this promise to it that it's going to deliver on the touch interface. You're going to be able to touch everything, and it's going to do what you expect," says engineer Ben Cunningham. "But there's more to it than that. There's this interface without any touch, the ability to move things by tilting back and forth."

But there are drawbacks to such a radical design. "The work is fascinating from a visual design perspective. It's a joy to play around with and explore. It certainly pushes the envelope in terms of what people are doing with mobile apps," says Greg Raiz, an ex-Microsoftie who now runs a San Francisco mobile development firm called Raizlabs. "But I think there are some usability issues with it. It's somewhat confusing." And indeed, if you look at the download numbers at the Apple App Store, the app has yet to attract a huge audience.

But in terms of design, apps like the Flipboard news reader are moving in a similar direction, and others believe this is just a first step toward a world where apps are move closely tied to our movements. "Heavily graphical and slick animations are far more important now," says Robert Armstrong, who works with a Southern California mobile development house called Appstem.

Facebook is certainly doing its part to fast-track this evolution. In addition to open-sourcing Origami, the team says it could do the same with Tweaks as well as the engine it built to more effectively spread tasks across multiple processor cores. It's also open-sourcing smaller bits of software used to build the app. This includes a software library called Shimmer, a widget built by 18-year-old Grant Paul that replaces the ubiquitous Apple spinning pinwheel with a more natural and less obtrusive way of indicating that an app is stalled while processing.

For the initiated, open-source software can be a strange animal. It may seem odd that Facebook would give away tools that could potentially provide an edge over competitors if kept in-house. But Facebook, like so many other companies, believes there's more value in letting others take its tech, push it in new directions, and perhaps even help Facebook improve its own tools. "A lot of the most modern, new creations in software could not have been built without the building blocks provided by the open-source community. We feel an obligation to contribute to this," Goodson says. "And we know that, ultimately, we are likely to benefit as well."

Raizlabs and Appstem are already taking a look at Origami, and though Raiz and Armstrong are unsure whether they can make use of the tool, they see the potential in not only Origami but Tweaks and the other tools Facebook invented in the process of creating Paper. These tools show how change now happens in the software world. "If a company like Facebook releases stuff like this," Armstrong says, "people will absolutely try to use them to their advantage." In other words, Paper could spawn something better than Paper.