Revealed at Last: Magic Leap's Vision for Augmented Reality, in 32 Patent Illustrations

Here's why Google invested $542 million in an secretive Florida startup.

A new patent application titled Planar Waveguide Apparatus with Diffraction Element(s) and System Employing Same sounds like a scientific snoozefest, but just also might provide a playbook for the next decade of interaction design.

The surprisingly broad patent application was filed by Magic Leap, the secretive, Florida-based "Cinematic Reality" startup that recently received $542 million dollars of venture capital from Google, Legendary Entertainment, and Andresseen Horowitz. And its 180 pages represent the first detailed depiction of how the augmented-reality company believes we'll use this mind-bending hardware.

Magic Leap has been secretive about how their system works technically, but a plethora of disclosures in their filings provide the broad outline. A lightweight head-mounted device will house a tiny projector comprised of bespoke prisms and lenses that will beam images onto the user’s retinas creating a "dynamic digitized light field signal." Apps, powered by mobile devices or body worn computers, will generate a steady stream of fantastic creatures and surreal tableaus delivered with stereo speakers and at 60 frames per second. Infrared positioning cameras, GPS modules, and multi-axis accelerometers will blend these otherworldly images with more banal surroundings of a basement man cave.

The big question is how we'll interact with devices like this. Sitting in front of a keyboard defeats the purpose. Voice commands like those used with Google Glass are clunky, poorly suited for public use, and inefficient. Even Oculus, the company that captured the hearts and eyes of VR fanatics, has said that they may not include native input support for the first few generations of their system.

What's the Future of UI? Just One Word: Totems

Magic Leap believes the future of user interface will come in the form of ordinary objects imbued with virtual, interactive powers. Described as "Totems," these are tangible objects that combine physical craftsmanship and computational projection. Virtual worlds and physical objects are melded in the wearer's retina. Put another way, with Magic Leap UI buttons are in the eye of the beholder.

The simplest embodiment of this concept is a "computer mouse," which could actually be any block of plastic, wood, metal, or glass. Rather than relying on sensors built into the device, front-facing cameras on the Magic Leap headset could theoretically track any piece of material that's been defined as a "mouse." The Magic Leap headset would constantly be plotting it's position, direction, distance, speed, and acceleration while feeding those cues back to the augmented reality display.

Another illustration from the patent document shows a keyboard made from a soft rubber that would deform and provide haptic feedback while the head mounted display overlaid images of the characters on it in the user's eye.

Some of the totem concepts are more elegant. Illustrations show wearers using the popular pinch to zoom gesture to create a "screen" on an odd-sized piece of metal. The concept is described in detail here: "The totem may simply provide a physical surface, and the AR system may render a user interface so as to appear to a user to be on one or more surfaces of the totem. For instance, the AR system may render a virtual computer keyboard and virtual trackpad to appear on a surface of a thin rectangular plate of aluminum which serves as a totem."

Some of the embodiments are surprisingly low tech, like a cube where each face has a popular website overlaid on it. The user would switch between apps by rotating this cube. A bracelet offers similar app switching behavior, but requires the user to caress a charm emblazoned with the app's logo.

The presence of the registered trademark "Sensorywear®" in the application, used to describe augmented reality devices, suggests that Magic Leap believes wearables will be a big part of the future of their product line. Copious images of gloves, watches, and rings in the patent drawings back up that conjecture.

In addition to these forward-looking totems there will also be traditional gesture-based controls, like those popularized by the Leap Motion sensor and Microsoft Kinect. The patent illustrates a series of motions that could turn any flat surface into an ad hoc workspace, described here: "The user may, for example, perform a two finger gesture, for instance an expanding pinch gesture, dragging outward from a center point to locations where an upper left and a lower right corner of the virtual work portal or construct should be located."

Gamer's have no reason to fear, it appears that traditional game controllers will also be supported. Other hardware concepts seem to be pushing into the world of concept art, like a piece of hardware that's shaped like a lotus and presumably treats each petal as an input method.

Otherworldly Tech, Yawn-Worthy Use Cases

There's a stark difference between the applications depicted on the Magic Leap website and in their patent filing.

Illustration: Magic Leap

The most surprising thing about the Magic Leap patent is how pedestrian most the applications for this breakthrough technology appear to be.

On the Magic Leap website, the uses cases are presented as equal parts Silicon Valley and Salvador Dalí. Whales list lazily over Miami Beach while dragons populate the sky. Most of the drawings in the patent application make Magic Leap look like a fancy replacement for universal remote controls.

A dozen illustrations show a guy in his basement interacting with a football game in novel ways. There is the de rigueur drawing of augmented reality being used in surgery. Exercise bikes will be made more interactive by creating a virtual Tour de France. Physical desktops will take on the interactive properties of their computation counterparts. These aren't bad applications, just similar to what was promised by Google Glass.

One of the more interesting aspects of the application is the surprising specificity and highly branded nature of the drawings. Grocery shoppers would be treated to a virtual representation of Mario Batali, not some stock art chef. A ghostly avatar of Toucan Sam would personally greet kids by name in the cereal aisle, rather than some off-brand spokesanimal. Twitter and Facebook logos are emblazoned on physical icons. This makes the drawings easier to parse, but also seems to play into the technical implementation of the system.

The patent calls out computer vision as a key component of many of these UI concepts and aims to solve the tricky challenges it presents using "Object Recognizers." These would be chunks of code that make it easy for Magic Leap software to identify a specific trademark, like the Starbucks' logo, in a crowded field of view. This has the side benefit of making the tool work better while also potentially strengthening the company's business model.

Many have posited uses for AR and VR, but few have illustrated them with the verve and detail that Magic Leap has. The breadth and detail of the patent's claims make the system seem like science fiction, but Magic Leap founder Rony Abovitz previously developed FDA-cleared has previously developed FDA-cleared surgical robots that he likened to the Droids from Star Wars. He sold that company, with a similarly futuristic premise, for $1.65 billion dollars, so you might say the force is strong with this one.