Leap Motion's Augmented-Reality Computing Looks Stupid Cool

A team of Leap Motion engineers built an augmented-reality work environment, where regular desktop applications jump out of the computer and into 3-D space.

We have a habit of filling new technologies with old ideas. Television borrowed from vaudeville in its early days. Personal computers were built around familiar pre-digital concepts like inboxes, file folders, and desktops. So before coming technologies like augmented and virtual reality give us entirely new worlds to explore, we can expect them to extend and expand the one we already know. This demo, in which a standard desktop computer is reimagined as a three-dimensional workstation of the future, offers a glimpse of what that might look like.

The project came out of a hackathon at Leap Motion, whose nifty gesture-recognizing sensor acts as a sort of finger-scale Kinect for desktop software. Using a prototype Leap sensor, a developer-kit Oculus Rift, a team of engineers built an augmented-reality work environment in which regular desktop applications jump out of the computer and into 3-D space. It's a new computing interface hovering in front of a traditional personal computer sitting on a wood table---three generations of the "desktop," one on top of another.

In the short POV clip, we swipe down a transparent, floating to-do list---then plunge our hand through it, like a movie character encountering a ghost for the first time. After that, we yank programs from our desktop displays into the space in front of them, rearranging and resizing the floating windows with our hands. All the while, small circles hover near the bottom of our field of vision---shortcuts for chatting with coworkers or friends.

The whole scene has a certain Minority Report flair, which is equal parts irresistible and impractical. It may look cool, sure, but no one is going to want to wave their hands around at work all day.

Still, having dipped his fingers into the 3-D computing future, Leap software engineer Raffi Bedikian came out convinced it could be tremendously useful for some applications."One thing we've learned is that humans are really quite good at spatially arranging things, categorizing them, and remembering where they were," he says. "For example, when browsing the web today, your tabs are organized in a list. What if they were clustered by topic, so that you might have your work-related tabs on the left side and your vacation-planning tabs on the right? And what if you could dynamically re-cluster the windows based on other criteria, like how long you've spent looking through them or relevance to a particular search query?" By combining traditional desktop OS concepts and the possibilities of augmented reality, designers might be able to better leverage innate human capabilities such as this spatial awareness, Bedikian says.

Bedikian and his team imagined the augmented-reality headset not as a replacement for today's computers but as a supplement to them. Keyboard and mouse still drive much of the input. The standard desktop user interface is retained for existing applications, with other menus and tool bars added for new functionality. (For the demo, the floating windows were actually linked up live to the desktop computer you see in the clip; the tool bars were placeholder art.) There are shortcomings with the setup, Bedikian says. For instance, due to the resolution of the current generation of head-mounted displays, it can be hard to read small text.

Still, he thinks this combined approach could be extremely powerful. He sees particular potential for skilled users, who could create 3-D workspaces tailored to their tasks. "Let's say I want to do something like accounting," Bedikian says. "With one interaction of my accounting bubble floating in space, I could load up a whole suite of relevant things. Maybe Excel would load in the middle, my email and cloud drive stacked on the left, and my useful macros and documentation on the right. Instead of manually maximizing and minimizing individual programs and windows, this would treat them as one unit and remember your favorite spatial arrangements of things. Powerful data visualization and interaction capabilities will have a profound impact on almost every use we've come up with for computers."

Another small feature in the video suggests other interesting possibilities for this sort of technology. It shows up when Bedikian starts watching a video in a hovering YouTube window. As it plays, he taps a button on a floating tool bar and sends the display into a virtual theater mode, blacking out the live feed of the real world around him. The team came up with this feature halfway through the hackathon, Bedikian says. It's a small thing, but it opens some compelling territory for designers to explore: the places where augmented reality and virtual reality can seamlessly intermingle.

For one example of how this might be useful, think about a long flight, Bedikan says. "You could toggle the passthrough on when the flight attendant is handing you your drink, then toggle it off again to resume your immersive 3-D movie experience."

Beyond just blocking out reality, though, future headsets that combine augmented reality and virtual reality might also be able to replace reality outright. They could bring your computer programs into 3-D space while transforming that space into something else. Think about a headset that could hide the distractions of your real-world office behind the virtual facade of a law library or a zen garden. "Can you imagine how awesome it would be if at the push of a button you could fly your virtual workspace on top of a mountain or in the middle of a lush forest?" Bedikian asks. A press of a button could send you on an instant commute to any virtual workspace of your choosing. Another way of thinking about it, Bedikian says: a really extreme version of changing your desktop wallpaper.